Tag : python

What is the order in which classes are instantiated in PythonGenerally speaking, there are class variables and methods in a class. For example, we define a class named A. class A(): bar = “my lover love me” def __init__(self, name): print(‘A的class’ ,self.__class__, name) We have defined a class variable bar and a construction method in ..

Read more

Design pattern learning 1. Decorator mode This is introduced in my notes. 2. Factory model author: liibntime:2018-11-6Not completed, to be updated 3. singleton pattern Ensure that at most one instance of an object exists. If there are multiple statements that create an instance, only the first one will actually create an instance, and the following ..

Read more

Python Multiprocess, Global Variable Not Updated Problem: Sub-process global variables are not updated as expected, resulting in data not being written to disk. The process pool must be in theif __name__ == ‘__main__’The following is usedReproduction: see the experimental code.Environment: centos7, python3.7Reason: of the subprocess__name__Attribute value is not’__main__’Solution: The resources needed should be inif __name__ ..

Read more

Oozie creates workflow The execution command of the workflow refers to the blog:https://www.jianshu.com/p/6cb3a4b78556, you can also typeoozie helpView help Manually configure oozie’s workflow Job.properties file, which stores some parameters that may be used in workflow.xml filejob.properties # 注意变量名不要包含特殊字符,否则在 spark 中会出现无法解析变量名的问题 # oozie.wf.application.path的路径必须在hdfs上,因为整个集群要访问 nameNode=hdfs://txz-data0:9820 resourceManager=txz-data0:8032 oozie.use.system.libpath=true oozie.libpath=${nameNode}/share/lib/spark2/jars/,${nameNode}/share/lib/spark2/python/lib/,${nameNode}/share/lib/spark2/hive-site.xml oozie.wf.application.path=${nameNode}/workflow/data-factory/download_report_voice_and_upload/Workflow oozie.action.sharelib.for.spark=spark2 archive=${nameNode}/envs/py3.tar.gz#py # 如果 dryrun 为 true,表示只是测试当前的 workflow,并不具体记录相应 ..

Read more

Python3.7 dictionary is orderly Old structure The dictionary structure before python3.7, the classic and crude hash table implementation, in this way every expansion and contraction of the hash table may cause the hash value to change. Before and after the hash table capacity update, the relative order between its keys will change, so the elements ..

Read more

Learn about Golang’s market situation Project address:https://github.com/go-crawler … If it is helpful to you, welcome Star and give the article a praise so that more people can see it:) Target Golang is already a part of the work. I want everyone to know about the market situation of Golang, and I also want more people ..

Read more

Fake-useragent of Go Some websites will jump to different (PC, M) websites according to the different User-Agent, and will give different prompts according to different versions, etc. The change of User-Agent is the basic posture in the crawler. If you use Go to write a web crawler or need to simulate a User-Agent, will it ..

Read more

About elastalert Elastalert is an elasticsearch-based alert service produced by yelp and written in python. The overall idea is still based on polling method. If rules are applied, frequency, spike, flatline, blacklist/whitelist, any and change are built in. In case of alarm, Email, HipChat, Slack, Telegram, etc. are provided. dockerfile # Elastalert Docker image running ..

Read more

example #coding=UTF-8 import sys import MySQLdb import time reload(sys) sys.setdefaultencoding(‘utf-8’) def connectDemo(): return MySQLdb.Connection(“127.0.0.1″,”root”,”root”,”demo”,3306,charset=”utf8″) if __name__ == ‘__main__’: begin=time.time() conn=connectDemo() cursor = conn.cursor() sql=””” show tables “”” count = cursor.execute(sql) rows = cursor.fetchall() cursor.close() conn.close() print “========demo库共:%s 张表============” % (count) print ‘耗时:%s 秒’ % (time.tim..

Read more