Wes Mckinney’s Ibis, a Pythonic interface to Impala, has functionality for creating Impala tables from Python Pandas dataframes.
1
2
3
4
5
6
7
8
9
| import pandas as pd
import ibis
hdfs = ibis.hdfs_connect(host=webhdfs_host, port=webhdfs_port)
client = ibis.impala.connect(host=impala_host, port=impala_port,
hdfs_client=hdfs)
db = c.database('ibis_testing')
data = pd.DataFrame({'foo': [1, 2, 3, 4], 'bar': ['a', 'b', 'c', 'd']})
db.create_table('pandas_table', data)
|
This functionality, added in Ibis 0.6.0, is much easier that manually move data to HDFS and loading into Impala.
