Skip to content

Commit 131c4c5

Browse files
rkrishn7rkrishn7
committed
update config docs
1 parent f429da1 commit 131c4c5

3 files changed

Lines changed: 130 additions & 0 deletions

File tree

‎cmds‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
set datafusion.execution.parquet.bloom_filter_on_read = true;
2+
set datafusion.execution.parquet.pushdown_filters = true;
3+
4+
CREATE EXTERNAL TABLE orders
5+
STORED AS PARQUET
6+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/orders/';
7+
8+
CREATE EXTERNAL TABLE customer
9+
STORED AS PARQUET
10+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/customer/';
11+
12+
CREATE EXTERNAL TABLE lineitem
13+
STORED AS PARQUET
14+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/lineitem/';
15+
16+
CREATE EXTERNAL TABLE partsupp
17+
STORED AS PARQUET
18+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/partsupp/';
19+
20+
CREATE EXTERNAL TABLE supplier
21+
STORED AS PARQUET
22+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/supplier/';
23+
24+
CREATE EXTERNAL TABLE nation
25+
STORED AS PARQUET
26+
LOCATION '/home/rohan/oss/datafusion/benchmarks/data/tpch_sf1/nation/';
27+
28+
select
29+
ps_partkey,
30+
sum(ps_supplycost * ps_availqty) as value
31+
from
32+
partsupp,
33+
supplier,
34+
nation
35+
where
36+
ps_suppkey = s_suppkey
37+
and s_nationkey = n_nationkey
38+
and n_name = 'GERMANY'
39+
group by
40+
ps_partkey having
41+
sum(ps_supplycost * ps_availqty) > (
42+
select
43+
sum(ps_supplycost * ps_availqty) * 0.0001
44+
from
45+
partsupp,
46+
supplier,
47+
nation
48+
where
49+
ps_suppkey = s_suppkey
50+
and s_nationkey = n_nationkey
51+
and n_name = 'GERMANY'
52+
)
53+
order by
54+
value desc;

‎docs/source/user-guide/configs.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ The following configuration settings are available:
7575
| datafusion.catalog.newlines_in_values | false | Specifies whether newlines in (quoted) CSV values are supported. This is the default value for `format.newlines_in_values` for `CREATE EXTERNAL TABLE` if not specified explicitly in the statement. Parsing newlines in quoted values may be affected by execution behaviour such as parallel file scanning. Setting this to `true` ensures that newlines in values are parsed successfully, which may reduce performance. |
7676
| datafusion.execution.batch_size | 8192 | Default batch size while creating new batches, it's especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption |
7777
| datafusion.execution.coalesce_batches | true | When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting |
78+
| datafusion.execution.hash_join_sideways_hash_passing | false | When set to true, hash joins will allow passing hashes from the build side to the right side of the join. This can be useful to prune rows early on, but may consume more memory. |
7879
| datafusion.execution.collect_statistics | true | Should DataFusion collect statistics when first creating a table. Has no effect after the table is created. Applies to the default `ListingTableProvider` in DataFusion. Defaults to true. |
7980
| datafusion.execution.target_partitions | 0 | Number of partitions for query execution. Increasing partitions can increase concurrency. Defaults to the number of CPU cores on the system |
8081
| datafusion.execution.time_zone | +00:00 | The default time zone Some functions, e.g. `EXTRACT(HOUR from SOME_TIME)`, shift the underlying datetime according to this time zone, and then extract the hour |

‎tmp‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
DataFusion CLI v50.0.0
2+
0 row(s) fetched.
3+
Elapsed 0.001 seconds.
4+
5+
0 row(s) fetched.
6+
Elapsed 0.000 seconds.
7+
8+
0 row(s) fetched.
9+
Elapsed 0.011 seconds.
10+
11+
0 row(s) fetched.
12+
Elapsed 0.008 seconds.
13+
14+
0 row(s) fetched.
15+
Elapsed 0.017 seconds.
16+
17+
0 row(s) fetched.
18+
Elapsed 0.007 seconds.
19+
20+
0 row(s) fetched.
21+
Elapsed 0.001 seconds.
22+
23+
0 row(s) fetched.
24+
Elapsed 0.001 seconds.
25+
26+
+------------+-------------+
27+
| ps_partkey | value |
28+
+------------+-------------+
29+
| 129760 | 17538456.86 |
30+
| 166726 | 16503353.92 |
31+
| 191287 | 16474801.97 |
32+
| 161758 | 16101755.54 |
33+
| 34452 | 15983844.72 |
34+
| 139035 | 15907078.34 |
35+
| 9403 | 15451755.62 |
36+
| 154358 | 15212937.88 |
37+
| 38823 | 15064802.86 |
38+
| 85606 | 15053957.15 |
39+
| 33354 | 14408297.40 |
40+
| 154747 | 14407580.68 |
41+
| 82865 | 14235489.78 |
42+
| 76094 | 14094247.04 |
43+
| 222 | 13937777.74 |
44+
| 121271 | 13908336.00 |
45+
| 55221 | 13716120.47 |
46+
| 22819 | 13666434.28 |
47+
| 76281 | 13646853.68 |
48+
| 85298 | 13581154.93 |
49+
| 85158 | 13554904.00 |
50+
| 139684 | 13535538.72 |
51+
| 31034 | 13498025.25 |
52+
| 87305 | 13482847.04 |
53+
| 10181 | 13445148.75 |
54+
| 62323 | 13411824.30 |
55+
| 26489 | 13377256.38 |
56+
| 96493 | 13339057.83 |
57+
| 56548 | 13329014.97 |
58+
| 55576 | 13306843.35 |
59+
| 159751 | 13306614.48 |
60+
| 92406 | 13287414.50 |
61+
| 182636 | 13223726.74 |
62+
| 199969 | 13135288.21 |
63+
| 62865 | 13001926.94 |
64+
| 7284 | 12945298.19 |
65+
| 197867 | 12944510.52 |
66+
| 11562 | 12931575.51 |
67+
| 75165 | 12916918.12 |
68+
| 97175 | 12911283.50 |
69+
| . |
70+
| . |
71+
| . |
72+
+------------+-------------+
73+
1048 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
74+
Elapsed 0.142 seconds.
75+

0 commit comments

Comments
 (0)