python requests 支持 socket 代理

让python requests 支持 socks5代理 requests support socket proxy 
pip install —upgrade requests “requests[socks]”

import requests
requests.get("http://www.google.com", proxies={"http":"sock5://127.0.0.1:1080"})

或者
pip install requests-unixsocket

import requests_unixsocket as requests
requests.get("http://www.google.com", proxies={"http":"sock5://127.0.0.1:1080"})

Composer 生产服性能优化指南

Level-1 优化:生成 classmap

生产环境一定要执行该命令,为啥呢?

composer dump-autoload -o(-o 等同于 --optimize

原理:

这个命令的本质是将 PSR-4/PSR-0 的规则转化为 classmap 规则(classmap 中包含了所有类名与类文件路径的对应关系)避免了加载器再去文件系统中遍历查找文件产生不必要的 IO。

当加载器找不到目标类时,仍旧会根据 PSR-4/PSR-0 的规则去文件系统中查找。

Level-2 优化:权威的(Authoritative)classmap

如果想实现加载器找不到类时即停止,那可以采用 Authoritative classmap:

composer dump-autoload -a-a 等同于 --classmap-authoritative

原理:

执行这个命令隐含的也执行了 Level-1 的命令, 即同样也是生成了 classmap,区别在于当加载器在 classmap 中找不到目标类时,不会再去文件系统中查找(即隐含的认为 classmap 中就是所有合法的类,不会有其他的类了,除非法调用)

注意: 如果你的项目在运行时会生成类,使用这个优化策略会找不到这些新生成的类。

Level-1 Plus 优化:使用 APCu Cache

在生产环境下,这个策略一般也会与 Level-1 一起使用, 执行:

composer dump-autoload -o --apcu

APCu 是 APC 去除 opcode 缓存后的精简版,只用于本机数据缓存(共享内存使得数据在多进程间可共享)。

这样,即使生产环境下生成了新的类,只需要文件系统中查找一次即可被缓存 , 弥补了 Level-2/A 的缺陷。

如何选择 & 小结

如果项目在运行时不会生成新的类文件,那么使用 Level-2/A,否则使用 Level-1 及 Level-1 Plus。

Level-2 的优化基本都是 Level-1 优化的补充,Level-2 主要是决定在 classmap 中找不到目标类时是否继续找下去。

Level-1 Plus 主要是在提供了一个缓存机制,将在 classmap 中找不到时,将从文件系统中找到的文件路径缓存起来,加速后续查找的速度。


Solving Error response from daemon OCI runtime create failed container with id exists

I restarted system for the testing my new systemd service. After the restart my some dockers start via command and I get this error.

root@fra ~>docker start bindnat64
Error response from daemon: OCI runtime create failed: container with id exists: 4871484679fsd498sg4hr9t989                                                                      9041a6b77dd7dc64b97e15cbd: unknown
Error: failed to start containers: bindnat64

For the detailed problem solving run docker in debug mode in command line.

root@fra ~> service docker stop
root@fra ~> dockerd --debug
INFO[2020-01-17T23:15:00.799572801Z] Starting up
DEBU[2020-01-17T23:15:00.816102056Z] Listener created for HTTP on unix (/var/run/docker.sock)
DEBU[2020-01-17T23:15:00.820722321Z] Golang's threads limit set to 2970
INFO[2020-01-17T23:15:00.836791146Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-17T23:15:00.837428925Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-17T23:15:00.838003954Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-17T23:15:00.838451356Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-01-17T23:15:00.859419482Z] parsed scheme: "unix"                         module=grpc
INFO[2020-01-17T23:15:00.860449827Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-01-17T23:15:00.861054713Z] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-01-17T23:15:00.861528201Z] ClientConn switching balancer to "pick_first"  module=grpc
DEBU[2020-01-17T23:15:00.871091213Z] Using default logging driver json-file
DEBU[2020-01-17T23:15:00.874882162Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2020-01-17T23:15:00.908944376Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2020-01-17T23:15:01.000878576Z] backingFs=extfs, projectQuotaSupported=false, indexOff="index=off,"  storage-driver=overlay2
INFO[2020-01-17T23:15:01.001615063Z] [graphdriver] using prior storage driver: overlay2
DEBU[2020-01-17T23:15:01.002094888Z] Initialized graph driver overlay2
***
ERRO[2020-01-17T23:15:02.591864507Z] stream copy error: reading from a closed fifo
ERRO[2020-01-17T23:15:02.592325661Z] stream copy error: reading from a closed fifo
***
INFO[2020-01-17T23:15:05.190269583Z] Loading containers: done.
INFO[2020-01-17T23:15:05.259382156Z] Docker daemon                                 commit=633a0ea838 graphdriver(s)=overlay2 version=19.03.5

😒

To solve this problem with remove container state folder inside docker folder.

#For example my container name bindnat64 and docker id is 4871484679fsd498sg4hr9t989.
root@fra ~> rm -rf /var/run/docker/runtime-runc/moby/4871484679fsd498sg4hr9t989
root@fra ~>docker start bindnat64
bindnat64

Looks like a problem solved but it`s not solved. When you try to execute program inside docker container you get another Error.

Close dockerd and restart your Server.

Hopely after the restarting system your docker container will be work without error.


Edit

In my case, I see this error again on my system. This time, it is a bit harder to fix then before. Because now it is not based on one container. My chance is all my containers is not important. So I deleted all stat information on a system.

root@fra ~> rm -rf /var/run/docker/runtime-runc/moby/*
root@fra ~> service docker stop
root@fra ~> reboot

I restart my server and delete stat information, I do this process until docker service works without any problem.


How the full page cache works in Magento 2

This Magento 2 tutorial looks at the Magento 2 full page cache (FPC). The FPC is a key performance feature in Magento 2, but differs significantly from the Magento 1 FPC.

One of the most notable aspects of FPC in Magento 2 is that it is now a standard feature in the Community Edition (CE). In Magento 1 FPC was an Enterprise Edition (EE) feature only, which meant that CE users had to buy a module to get this sort of functionality.

The other equally notable aspect of Magento 2’s FPC is that Varnish integration is available in FPC with only minimal configuration, which means that caching is at the heart of Magento 2 rather than an afterthought.

Changes at a glance

Magento 1 Magento 2
EE only feature CE and EE feature
Varnish distance from FPC Varnish integrated
Built-in option only Built-in and Varnish options
Built-in recommended for production Varnish recommended for production
Hole punching for private content Placeholders and AJAX/local storage for private content
Javascript used as a workaround to deliver private content Javascript integral to delivering private content

The built-in FPC option

(continue reading…)


library not found for -ljanalytics-ios-2.1.2

需要把
/node_modules/janalytics-react-native/ios/RCTJAnalyticsModule/janalytics-ios-2.1.2.a
文件名修改为
libjanalytics-ios-2.1.2.a

 


React/Core error

grep -rl "s.dependency 'React/Core'" node_modules/ | xargs sed -i '' 's=React/Core=React-Core=g'

 


docker 启动容器遇到问题: cgroups: cannot find cgroup mount destination: unknown

解决办法:
1、执行Linux虚拟机。
2、执行 sudo mkdir /sys/fs/cgroup/systemd
3、执行 sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
4、再次启动容器,如下图:

 


Solving Error response from daemon OCI runtime create failed container with id exists

Backup then:
root@aaa ~> rm -rf /var/run/docker/runtime-runc/moby/*
root@aaa ~> service docker stop
root@aaa ~> reboot

 

问题发生
有两个闲置的容器在删除时发生错误,处在Removal In Progress状态。在重启Docker后处于Dead状态。

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                PORTS                                      NAMES
cdcf0acae1af        nginx:latest        "nginx -g 'daemon of…"   6 months ago        Removal In Progress                                              webproxy
c58e2a73f117        hexoblogs:latest    "nginx -g 'daemon of…"   11 months ago       Removal In Progress                                              dhblogs
删除时提示信息如下:


$ sudo docker rm webproxy
Error response from daemon: driver "devicemapper" failed to remove root filesystem for cdcf0acae1afb7da7e57653945b7322e32bc6909444558ba6834612b8bfa5ebd: remove /mnt/docker/devicemapper/mnt/5faf8f2ec431bad0e1e2938330b2844209e864b1b76c857fea5f29b44a18999a: device or resource busy
网上有很多类似的解决办法,但试了几个均不能使用,例如:

$ docker system prune
$ find /var/lib/docker -name "*webproxy*"
$ mount -l
$ umount
$ ...
最终解决方案:

$ grep docker /proc/*/mountinfo | grep 5faf8f2ec431bad0e1e2938330b2844209e864b1b76c857fea5f29b44a18999a
/proc/24762/mountinfo:145 143 252:5 / /mnt/docker/devicemapper/mnt/5faf8f2ec431bad0e1e2938330b2844209e864b1b76c857fea5f29b44a18999a rw,relatime shared:97 - xfs /dev/mapper/docker-253:17-15728997-5faf8f2ec431bad0e1e2938330b2844209e864b1b76c857fea5f29b44a18999a rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota
$ kill -9 24762
$ docker rm dhblogs
dhblogs
说明:grep检索的不是容器号,而是删除时提示信息中/mnt/docker/devicemapper/mnt/*** 驱动器号,得到占用该驱动器的进程号 24762(/proc/24762/mountinfo),杀死进程即可

 

 

 


安装多个mysql

变异安装mysql:
https://dev.mysql.com/doc/refman/5.7/en/installing-source-distribution.html
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.30.tar.gz

shell> mkdir bld shell> cd bld
cmake .. -DMYSQL_TCP_PORT=3307 -DMYSQL_UNIX_ADDR=/var/run/mysqld/5.7/mysql.sock -DCMAKE_INSTALL_PREFIX=/usr/local/mysql/5.7 -DMYSQL_DATADIR=/home/mysql5.7 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/usr/local/boost
shell> make shell> make install

bin/mysqld --defaults-file=/etc/mysql/5.7/my.cnf --initialize --user=mysql
=======start========= 
chown -R mysql:mysql /var/run/mysqld/5.7 
/usr/local/mysql/5.7/bin/mysqld_safe --defaults-file=/etc/mysql/5.7/my.cnf

/usr/local/mysql/5.7/bin/mysql -h127.0.0.1 -P 3307 -uroot -p
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root-password';
CREATE USER 'root'@'%' IDENTIFIED BY 'root-password';
grant all on *.* to 'root'@'%' with grant option;

 

-----------------
~# cat /etc/my.cnf
[mysqld]
user		= mysql
port		= 3307
basedir		= /usr/local/mysql/5.7
pid-file	= /var/run/mysqld/5.7/mysqld.pid
socket		= /var/run/mysqld/5.7/mysqld.sock
datadir		= /var/lib/mysql/5.7
log-error	= /var/log/mysql/error-5.7.log
# By default we only accept connections from localhost
#bind-address	= 127.0.0.1
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

query_cache_size=512M
query_cache_type=1
key_buffer_size=256M
innodb_buffer_pool_size=512M

language = /usr/local/mysql/5.7/share/english


sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION


 

配置开机启动项

cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
chmod +x /etc/init.d/mysqld
chkconfig --add mysqld
chkconfig mysqld on

爬虫网页

#nohup scrapy runspider spider.py > urls.log &
import scrapy

class MySpider(scrapy.Spider):
    name = 'domain'
    allowed_domains = ['domain']
    start_urls = [
        'https://domain/'
    ]

    #avoid repeat
    tempList = []

    def parse(self, response):
        # for h3 in response.xpath('//h3').getall():
        #     yield {"title": h3}

        for link in response.xpath('//a/@href').getall():
            if link not in self.tempList:
                print(link)
                self.tempList.append(link)
                yield scrapy.Request(response.urljoin(link), self.parse)

 


Copyright © 1996-2010 Add Lives. All rights reserved.
iDream theme by Templates Next | Powered by WordPress