Abort fork on intermediate states #2043

AviAvni · 2021-11-24T13:59:53Z

No description provided.

codecov · 2021-11-24T16:18:10Z

Codecov Report

Merging #2043 (4db62fc) into master (53ddab0) will increase coverage by 0.06%.
The diff coverage is 96.29%.

@@            Coverage Diff             @@
##           master    #2043      +/-   ##
==========================================
+ Coverage   92.77%   92.83%   +0.06%     
==========================================
  Files         247      248       +1     
  Lines       21648    21666      +18     
==========================================
+ Hits        20084    20114      +30     
+ Misses       1564     1552      -12

Impacted Files	Coverage Δ
...rc/serializers/decoders/current/v10/decode_graph.c	`96.96% <ø> (-0.07%)`	⬇️
src/serializers/decoders/prev/v7/decode_graph.c	`96.47% <ø> (-0.09%)`	⬇️
src/serializers/decoders/prev/v8/decode_graph.c	`96.73% <ø> (-0.07%)`	⬇️
src/serializers/decoders/prev/v9/decode_graph.c	`96.93% <ø> (-0.07%)`	⬇️
src/module.c	`71.92% <50.00%> (-0.80%)`	⬇️
src/module_event_handlers.c	`97.74% <94.73%> (+8.35%)`	⬆️
src/commands/cmd_debug.c	`100.00% <100.00%> (ø)`
src/graph/graphcontext.c	`97.38% <100.00%> (-0.02%)`	⬇️
src/serializers/decode_context.c	`93.02% <100.00%> (+0.16%)`	⬆️
src/serializers/graphcontext_type.c	`89.18% <100.00%> (+5.31%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53ddab0...4db62fc. Read the comment docs.

src/commands/cmd_debug.c

src/serializers/graphcontext_type.c

src/module_event_handlers.c

swilly22 · 2021-11-25T08:56:36Z

src/module_event_handlers.c

-		_ResetDecodeStates();
+		uint count = array_len(graphs_in_keyspace);
+		for (size_t i = 0; i < count; i++) {
+			GraphContext_Delete(graphs_in_keyspace[i]);


Can you please explain how the ref count is managed in case where the original graph key is present in the key-space and when the original graph key is missing ?

in that case when redis call the free function it supply the freed graph context
there is nothing to remove from registry and the refcount going to be decreased to -2 and to not free the graph context

src/module_event_handlers.c

swilly22 · 2021-11-25T08:59:52Z

src/graph/graphcontext.c

@@ -34,7 +33,7 @@ static inline void _GraphContext_IncreaseRefCount(GraphContext *gc) {

 static inline void _GraphContext_DecreaseRefCount(GraphContext *gc) {
 	// If the reference count is less than 0, the graph has been marked for deletion and no queries are active - free the graph.
-	if(__atomic_sub_fetch(&gc->ref_count, 1, __ATOMIC_RELAXED) < 0) {
+	if(__atomic_sub_fetch(&gc->ref_count, 1, __ATOMIC_RELAXED) == -1) {


Should we add: ASSERT(gc->ref_count > -2); ?

before or after?

I don't think so in our case we can get to -2 but want to call free once

src/serializers/graphcontext_type.c

swilly22 · 2021-11-28T09:42:40Z

src/commands/cmd_debug.c

+extern GraphContext **graphs_in_keyspace;
+extern uint aux_field_counter;
+
+static void Debug_AUX(const char *arg) {


pass original argv and argc you might want to pass argv+1, argc-1 as the first value AUX was consumed.

swilly22 · 2021-11-28T09:50:51Z

tests/flow/test_replication_states.py

+
+                self._connection_permutation(connection_permutation, 3)
+
+                self._step(scenario[2], 3, None)


remove last argument, always None

swilly22 · 2021-11-28T12:29:57Z

src/commands/cmd_debug.c

@@ -16,7 +16,11 @@ void ModuleEventHandler_AUXAfterKeyspaceEvent(void);
 extern GraphContext **graphs_in_keyspace;
 extern uint aux_field_counter;

-static void Debug_AUX(const char *arg) {
+static void Debug_AUX(RedisModuleString **argv, int argc) {
+	if(argc < 2) return;


@AviAvni Emit an error in this case

swilly22 · 2021-11-28T12:30:32Z

@AviAvni Cherry pick into 2.4

* remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]>

* Abort fork on intermediate states (#2043) * remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> * update keys Co-authored-by: swilly22 <[email protected]>

* remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> (cherry picked from commit ecd0552)

* Abort fork on intermediate states (#2043) * remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> (cherry picked from commit ecd0552) * bump version to 2.8.7 * Test against RE version 100.0.0-2646 Co-authored-by: Avi Avni <[email protected]>

* remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]>

* index data on key load * remove index at end * index costruct both indices * Abort fork on intermediate states (#2043) * remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> * update keys * filter index scan result when index can't be utilize properly * fix test * handle reset * add more tests * make vkey_entity_count runtime config * address review * add tests * address review * fix test * address review Co-authored-by: Avi Avni <[email protected]>

* Index on vkey (#2067) * index data on key load * remove index at end * index costruct both indices * Abort fork on intermediate states (#2043) * remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> * update keys * filter index scan result when index can't be utilize properly * fix test * handle reset * add more tests * make vkey_entity_count runtime config * address review * add tests * address review * fix test * address review Co-authored-by: Avi Avni <[email protected]> * fix merge * fix merge * fix merge * fix merge * fix merge * fix merge * fix merge * refine * address edge index scan review * fix checking the accuracy for index * order by index scan results * add comments * clean decode_graph * a bit more refactoring around encoder/decoder * address encoder/decoder review * clean * clean decode graph schema * address review * remove paren Co-authored-by: Roi Lipman <[email protected]> Co-authored-by: swilly22 <[email protected]>

* remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]>

* Index on vkey (RedisGraph#2067) * index data on key load * remove index at end * index costruct both indices * Abort fork on intermediate states (RedisGraph#2043) * remove currently decoding graphs * removed AOF rewrite callback * abort encoding if intermediate graph detected * handle intermediate at the fork callback level * removed currently_decoding_graphs from graphcontext * add debug command * clear graphs_in_keyspace in flush all * add replication states tests * move exit to aux save * move exit to persistence event handler * address review * remove aux_field_counter * address review * address review Co-authored-by: swilly22 <[email protected]> * update keys * filter index scan result when index can't be utilize properly * fix test * handle reset * add more tests * make vkey_entity_count runtime config * address review * add tests * address review * fix test * address review Co-authored-by: Avi Avni <[email protected]> * fix merge * fix merge * fix merge * fix merge * fix merge * fix merge * fix merge * refine * address edge index scan review * fix checking the accuracy for index * order by index scan results * add comments * clean decode_graph * a bit more refactoring around encoder/decoder * address encoder/decoder review * clean * clean decode graph schema * address review * remove paren Co-authored-by: Roi Lipman <[email protected]> Co-authored-by: swilly22 <[email protected]>

swilly22 and others added 9 commits November 24, 2021 09:56

remove currently decoding graphs

3538cfd

removed AOF rewrite callback

bc90346

abort encoding if intermediate graph detected

2e0c2de

handle intermediate at the fork callback level

94970ca

removed currently_decoding_graphs from graphcontext

07f493e

add debug command

619108c

clear graphs_in_keyspace in flush all

3dc4ad3

add replication states tests

99a4166

move exit to aux save

d436981

move exit to persistence event handler

0d5cfd2

swilly22 suggested changes Nov 25, 2021

View reviewed changes

address review

3040c37

swilly22 suggested changes Nov 25, 2021

View reviewed changes

src/serializers/graphcontext_type.c Outdated Show resolved Hide resolved

AviAvni added 2 commits November 25, 2021 13:25

remove aux_field_counter

f1925b3

address review

41d917e

swilly22 suggested changes Nov 28, 2021

View reviewed changes

address review

4db62fc

swilly22 approved these changes Nov 28, 2021

View reviewed changes

swilly22 merged commit ecd0552 into master Nov 28, 2021

swilly22 deleted the abort-fork-on-inter branch November 28, 2021 12:30

chayim mentioned this pull request Jun 23, 2022

Fixing docker builds in 2.4.x #2426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Abort fork on intermediate states #2043

Abort fork on intermediate states #2043

AviAvni commented Nov 24, 2021

codecov bot commented Nov 24, 2021 •

edited

Loading

swilly22 Nov 25, 2021

AviAvni Nov 25, 2021

swilly22 Nov 25, 2021

AviAvni Nov 25, 2021

swilly22 Nov 25, 2021

AviAvni Nov 25, 2021

swilly22 Nov 28, 2021

swilly22 Nov 28, 2021

swilly22 Nov 28, 2021

swilly22 commented Nov 28, 2021


		self._connection_permutation(connection_permutation, 3)

		self._step(scenario[2], 3, None)

Abort fork on intermediate states #2043

Abort fork on intermediate states #2043

Conversation

AviAvni commented Nov 24, 2021

codecov bot commented Nov 24, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swilly22 commented Nov 28, 2021

codecov bot commented Nov 24, 2021 •

edited

Loading