Elasticsearch和Mysql的同步-logstash-input-jdbc

配置好elasticsearch后,需要将mysql的数据导入到elasticsearch中。网上查看,有好几种同步插件。我在elasticsearch-jdbc和 logstash-input-jdbc 中选择。两种我都尝试了下,由于我的elasticsearch版本是官网最新的6.1.1版本,官方现在推荐的都是logstash,于是把精力放在了logstash上。具体步骤如下:

1、logstash安装

Logstash是一款轻量级的日志搜集处理框架,可以方便的把分散的、多样化的日志搜集起来,并进行自定义的处理,然后传输到指定的位置,比如某个服务器或者文件。直接下载解压缩放在适当目录就行,免安装。下面是我的目录地址:

1
2
3
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.1.1.tar.gz
tar -zxvf logstash-6.1.1.tar.gz

Continue reading

Elasticsearch的安装部署和使用

最近由于工作之前遇到搜索的问题,于是开始研究Elasticsearch,准备运用到我们的系统中。于是先在macbook上安装实验,打算在一台机器上完成2个节点的配置。

一、Elasticsearch安装配置

1、下载安装

1
2
3
4
cd /usr/local/src
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.0.tar.gz
tar -zxvf elasticsearch-6.3.0.tar.gz
mv elasticsearch-6.3.0 /usr/local/

Continue reading

一次向svn中增加所有新增文件 svn add all new files【转】

我经常会一次往Subversion里添加一批文件。在使用命令行做这件事时,你必须指定所有想要添加的文件名。如果文件不多的话这还不算太糟糕,但如果你要添加20个文件,那就费事了。当然你也可以用通配符,但这样一来就可能匹配到已经在版本控制之下的文件(这不会有什么损害,只不过会输出一堆错误信息,可能会跟别的错误信息混淆)。为了解决这个问题,我写了一行简单的bash命令:

 

svn st | grep '^\?' | tr '^\?' ' ' | sed 's/[ ]*//' | sed 's/[ ]/\\ /g' | xargs svn add

Continue reading

使用Apollo 的订阅/发布 在 WebSockets 上轻松实现web页消息推送

最近要用到实时消息推送到web页面,从ajax轮询,长链接,到第三方的workman的webMsgSender都试了试。考虑到我们系统用到了apache apollo来进行消息的处理。而且apollo也支持websocket通讯,所以开始研了下,并测试运用成功!

开始之前,先了解下apollo的用户及安全策略

Authorization

因为要在前端使用用户名和密码,所以必须要做好安全策略。


1
2
3
4
5
6
<broker>
 <access_rule allow="admins" action="*"/>
 <access_rule allow="users" action="connect,receive"/>
 <access_rule allow="*" action="connect" kind="connector"/>
</broker>
这里面增加规则设置users组,当然前提是在users.properties和groups.properties里增加了组users和用户user,并给予action只connect,receive.

Continue reading

在网站上使用web-msg-sender进行实时消息推送

WEB消息推送

web-msg-sender是一款web长连接推送框架,采用PHPSocket.IO开发,基于WebSocket长连接通讯,如果浏览器不支持WebSocket则自动转用comet推送。 通过后台推送消息,消息可以即时推送到客户端,非轮询,实时性非常好,性能很高。

特点:

  • 多浏览器支持
  • 支持针对单个用户推送消息
  • 支持向所有用户推送消息
  • 长连接推送(websocket或者comet),消息即时到达
  • 支持在线用户数实时统计展示
  • 支持在线页面数实时统计展示
  • 支持跨域推送

Continue reading

wordpress错误ERR_INCOMPLETE_CHUNKED_ENCODING解决方法

在阿里云上,搭建了个nginx服务器,今天wordpress访问不正常,css没有加载成功,后台媒体库也加载不了。Chrome控制台看到如下错误:ERR_INCOMPLETE_CHUNKED_ENCODING。

看了下nginx/logs/error.log下日志:

  1. open() “/aliyun/server/nginx/fastcgi_temp/1/02/0000000021” failed (13: Permission denied) while reading upstream, client: x.x.x.x, server: x.x.x.x, request: “GET /blog/wp-admin/load-styles.php?c=1&dir=ltr&load=dashicons,admin-bar,wp-admin,buttons,wp-auth-check&ver=4.1.1 HTTP/1.1″, upstream: “fastcgi://127.0.0.1:9000”, host: “xx.xx.xx.xx”, referrer: “http://x.x.x.x/blog/wp-admin/”

搜索了下,找到这篇文章http://www.tuicool.com/articles/7Bjiyq跟我遇到的问题类似。

发现nginx目录下fastcgi_temp属于nobody用户和用户组。

Continue reading

安装使用 ActiveMQ Apollo

下载


1
2
3
4
5
6
7
<span class="token function">wget</span> http<span class="token punctuation">:</span><span class="token operator">/</span><span class="token operator">/</span><span class="token function">ftp</span><span class="token punctuation">.</span>kddilabs<span class="token punctuation">.</span>jp<span class="token operator">/</span>infosystems<span class="token operator">/</span>apache<span class="token operator">/</span>activemq<span class="token operator">/</span>activemq<span class="token operator">-</span>apollo<span class="token operator">/</span><span class="token number">1.7</span><span class="token punctuation">.</span>1<span class="token operator">/</span>apache<span class="token operator">-</span>apollo<span class="token operator">-</span><span class="token number">1.7</span><span class="token punctuation">.</span>1<span class="token operator">-</span>unix<span class="token operator">-</span>distro<span class="token punctuation">.</span><span class="token function">tar</span><span class="token punctuation">.</span>gz

<span class="token comment" spellcheck="true"># 将其解压到 /usr/local/activemq-apollo 目录
</span>
<span class="token comment" spellcheck="true"># 创建目录
</span><span class="token function">mkdir</span> <span class="token operator">/</span>usr<span class="token operator">/</span>local<span class="token operator">/</span>activemq<span class="token operator">-</span>apollo
<span class="token function">tar</span> xvzf apache<span class="token operator">-</span>apollo<span class="token operator">-</span><span class="token number">1.7</span><span class="token punctuation">.</span>1<span class="token operator">-</span>unix<span class="token operator">-</span>distro<span class="token punctuation">.</span><span class="token function">tar</span><span class="token punctuation">.</span>gz  <span class="token operator">-</span>C <span class="token operator">/</span>usr<span class="token operator">/</span>local<span class="token operator">/</span>activemq<span class="token operator">-</span>apollo <span class="token operator">--</span>strip<span class="token operator">-</span>components<span class="token operator">=</span><span class="token number">1</span>

创建 broker


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<span class="token comment" spellcheck="true"># 先创建broker存储目录
</span><span class="token function">mkdir</span> <span class="token operator">/</span>data<span class="token operator">/</span>apollo<span class="token operator">/</span>

<span class="token comment" spellcheck="true"># 创建broker
</span>$ <span class="token function">cd</span> <span class="token operator">/</span>usr<span class="token operator">/</span>local<span class="token operator">/</span>activemq<span class="token operator">-</span>apollo
$ bin<span class="token operator">/</span>apollo create <span class="token operator">/</span>data<span class="token operator">/</span>apollo<span class="token operator">/</span>broker
Creating apollo instance at<span class="token punctuation">:</span> <span class="token operator">/</span>data<span class="token operator">/</span>apollo<span class="token operator">/</span>broker
Generating ssl keystore<span class="token punctuation">.</span><span class="token punctuation">.</span><span class="token punctuation">.</span>

You can now start the broker by executing<span class="token punctuation">:</span>

   <span class="token string">"/data/apollo/broker/bin/apollo-broker"</span> run

Or you can setup the broker as system <span class="token function">service</span> and run it <span class="token keyword">in</span> the background<span class="token punctuation">:</span>

   <span class="token function">sudo</span> <span class="token function">ln</span> <span class="token operator">-</span>s <span class="token string">"/data/apollo/broker/bin/apollo-broker-service"</span> <span class="token operator">/</span>etc<span class="token operator">/</span>init<span class="token punctuation">.</span>d<span class="token operator">/</span>
   <span class="token operator">/</span>etc<span class="token operator">/</span>init<span class="token punctuation">.</span>d<span class="token operator">/</span>apollo<span class="token operator">-</span>broker<span class="token operator">-</span><span class="token function">service</span> start

配置web admin


1
<span class="token function">sed</span> <span class="token operator">-</span>i <span class="token string">'s/127.0.0.1:61680/192.168.2.11:61680/'</span> <span class="token operator">/</span>data<span class="token operator">/</span>apollo<span class="token operator">/</span>broker<span class="token operator">/</span>etc<span class="token operator">/</span>apollo<span class="token punctuation">.</span>xml

启动服务


1
2
<span class="token function">ln</span> <span class="token operator">-</span>s <span class="token string">"/data/apollo/broker/bin/apollo-broker-service"</span> <span class="token operator">/</span>etc<span class="token operator">/</span>init<span class="token punctuation">.</span>d<span class="token operator">/</span>
<span class="token function">service</span>  apollo<span class="token operator">-</span>broker<span class="token operator">-</span><span class="token function">service</span> start