从 PERL 脚本获取输出并将其加载到 MySQL 数据库的解决方案

news2025/7/15 4:29:41

在这里插入图片描述

1、问题背景

有一段 Python 脚本可以调用 Perl 脚本来解析文件，解析后，Perl 脚本会生成一个输出，这个输出将被加载到 Python 脚本中的 MySQL 数据库中。Python 脚本如下：

pipe = subprocess.Popen(["perl", "./parseGeneticCode.pl"], stdin=subprocess.PIPE)
pipe.stdin.close()
pipe = subprocess.Popen(["perl", "./makeTaxon.pl"], stdin=subprocess.PIPE)
pipe.stdin.close()

#Load taxon.out (output of makeTaxon.pl) in the database
sql10 = """LOAD DATA LOCAL INFILE 'gene_code.out' INTO TABLE geneticcode FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' (ncbi_taxon_ID, parent_ID, taxon_name, taxon_strain, rank, geneticcode_ID, mitochondrial_geneticcode_ID);"""
try:
  c.execute(sql10)
  conn.commit()
  print "\nDone uploading in taxon\n"
except StandardError, e:
  print e
  conn.rollback()
conn.close()

sql9 = """LOAD DATA LOCAL INFILE 'taxon.out' INTO TABLE taxon FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' (ncbi_taxon_ID, parent_ID, taxon_name, taxon_strain, rank, geneticcode_ID, mitochondrial_geneticcode_ID);"""
try:
  c.execute(sql9)
  conn.commit()
  print "\nDone uploading in taxon\n"
except StandardError, e:
  print e
  conn.rollback()
conn.close()

其中，文件 taxon.out 和 gene_code.out 由 Perl 脚本创建。运行此 Python 脚本时会报错：

(2, "File 'taxon.out' not found (Errcode: 2)")
(0, '')
Traceback (most recent call last):
  File "common_data.py", line 247, in <module>
    conn.rollback()
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

也就是说，文件 gene_code.out 和 taxon.out 都没有被上传到数据库中。

2、解决方案

在 Python 脚本中，使用 subprocess.Popen() 函数来调用 Perl 脚本时，需要在 Popen() 函数的 stdout= 参数中设置一个管道，以便将 Perl 脚本的输出重定向到该管道中。这样，就可以在 Python 脚本中读取 Perl 脚本的输出，并将其加载到 MySQL 数据库中。

修改后的 Python 脚本如下：

pipe = subprocess.Popen(["perl", "./parseGeneticCode.pl"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
pipe.stdin.close()
pipe_output = pipe.stdout.read()
#Load gene_code.out (output of parseGeneticCode.pl) in the database
sql10 = """LOAD DATA LOCAL INFILE 'gene_code.out' INTO TABLE geneticcode FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' (ncbi_taxon_ID, parent_ID, taxon_name, taxon_strain, rank, geneticcode_ID, mitochondrial_geneticcode_ID);"""
try:
  c.execute(sql10)
  conn.commit()
  print "\nDone uploading in taxon\n"
except StandardError, e:
  print e
  conn.rollback()
conn.close()

pipe = subprocess.Popen(["perl", "./makeTaxon.pl"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
pipe.stdin.close()
pipe_output = pipe.stdout.read()
sql9 = """LOAD DATA LOCAL INFILE 'taxon.out' INTO TABLE taxon FIELDS TERMINATED BY '\t' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' (ncbi_taxon_ID, parent_ID, taxon_name, taxon_strain, rank, geneticcode_ID, mitochondrial_geneticcode_ID);"""
try:
  c.execute(sql9)
  conn.commit()
  print "\nDone uploading in taxon\n"
except StandardError, e:
  print e
  conn.rollback()
conn.close()