MINOR: task: add a new explicitly local tasklet wakeup function

The current tasklet_wakeup() call relies on tasklet_wakeup_on(tl->tid),
which was already quite ambiguous till now due to the sole reliance on
tid being negative or not to decide to run locally, but it no longer
works correctly if used to wake tasks up since the new set of possible
negative values for ->tid (particularly if some code calls
__tasklet_wakeup_on() on a task as is done in task_instant_wakeup()).

The problem is that it is not possible in the current API to explicitly
say that we want a task/tasklet to run locally or remotely without having
to play games with a thread number. The chosen approach to address this
is to change tasklet_wakeup_on() to always be remote and have
tasklet_wakeup_here() which will always be local, with tasklet_wakeup()
choosing one or the other depending on the tid, for backwards compat
only.

This patch implements tasklet_wakeup_here() to __tasklet_wakeup_here()
that reimplement the part of __tasklet_wakeup_on() that used to deal
with the local thread only (negative tid). No other change was made.
For now it remains unused.

The doc was updated.

doc/internals/api/scheduler.txt

@@ -36,7 +36,7 @@ fields or flags may not always be relevant to tasklets and may be ignored.
 
 
 Tasklets do not use a thread mask but use a thread ID instead, to which they
-are bound. If the thread ID is negative, the tasklet is not bound but may only
+are bound. If the thread ID is -1, the tasklet is not bound but may only
 be run on the calling thread.
 
 A task with a negative thread ID is a shared task, and can run on any thread.
@@ -44,18 +44,26 @@ If its thread ID is -1, it means it is currently not owned by any thread. When
 a thread adds it to its timers queue, or its run queue, it will change the
 task's thread ID to -2 - calling thread ID, indicating that it now temporarily
 owns that task.
+
 If any other thread wants to have that task scheduled at a later time, or
 run, it will add it to that thread's run queue. If the goal is to add it to
 the timers queue, the TASK_WOKEN_WQ flag will be used, and the target thread
 will queue it to its timers queue.
+
 The task thread ID will be set to -1 by the owner thread when it is no longer
 in any queue, or running, indicating that that task no longer has any owner,
 and any thread can run it.
+
 To try to make sure a task won't stick on a thread forever, when a thread
 removes a task from its timers queue because it should run, if it detects
 that it has many more tasks than another random thread, it will just give
 the ownership to that other thread, and get the task to run on it.
 
+Tasklets do not use negative thread IDs to indicate a thread affinity, however
+since tasks may occasionally be aliased as tasklets, some of the tasklet
+handling functions take care of this specificity in order to preserve a full
+compatibility between them.
+
 
 2. API
 ------
@@ -134,17 +142,27 @@ struct list *tasklet_wakeup_after(head, tl, [flags])
         tasklet's flags, typically TASK_WOKEN_* or TASK_F_UEVT*. When not set,
         0 is passed (i.e. no flags are changed).
 
-void tasklet_wakeup_on(tl, thr, [flags])
-        Make sure that tasklet <tl> will wake up on thread <thr>, that is, will
-        execute at least once. The designated thread may only differ from the
-        calling one if the tasklet is already configured to run on another
-        thread, and it is not permitted to self-assign a tasklet if its tid is
-        negative, as it may already be scheduled to run somewhere else. Just in
-        case, only use tasklet_wakeup() which will pick the tasklet's assigned
+void tasklet_wakeup_here(tl, [flags])
+        Make sure that tasklet <tl> will wake up on the calling thread, that
+        is, will execute at least once. The tasklet must not be bound to any
+        other thread; its tid must be either -1 or the caller's tid. Just in
+        case, only use tasklet_wakeup() which will check the tasklet's assigned
         thread ID. An optional <flags> value may be passed to set a wakeup
         cause on the tasklet's flags, typically TASK_WOKEN_* or TASK_F_UEVT*.
         When not set, 0 is passed (i.e. no flags are changed).
 
+void tasklet_wakeup_on(tl, thr, [flags])
+        Make sure that tasklet <tl> will wake up on thread <thr>, that is, will
+        execute at least once. Any valid thread number may be specified. If the
+        tasklet is not yet queued, then it will be added to the designated
+        thread's shared queue. While this method also works for wakeups on the
+        calling thread, it's less efficient and whenever the calling thread
+        knows it will wake up one of its own tasks, it ought to use
+        tasklet_wakeup_here() or tasklet_wakeup() instead. An optional <flags>
+        value may be passed to set a wakeup cause on the tasklet's flags,
+        typically TASK_WOKEN_* or TASK_F_UEVT*. When not set, 0 is passed (i.e.
+        no flags are changed).
+
 struct tasklet *tasklet_new()
         Allocate a new tasklet and set it to run by default on the calling
         thread. The caller may change its tid to another one before using it.

include/haproxy/task.h

@@ -90,6 +90,7 @@ extern struct pool_head *pool_head_task;
 extern struct pool_head *pool_head_tasklet;
 extern struct pool_head *pool_head_notification;
 
+void __tasklet_wakeup_here(struct tasklet *tl);
 void __tasklet_wakeup_on(struct tasklet *tl, int thr);
 struct list *__tasklet_wakeup_after(struct list *head, struct tasklet *tl);
 void task_kill(struct task *t);
@@ -437,6 +438,42 @@ static inline void task_set_thread(struct task *t, int thr)
 	}
 }
 
+/* schedules tasklet <tl> to run on the current thread. Note that it is illegal
+ * call this with a task/tasklet that is neither agnostic to the running thread
+ * (->tid==-1) nor bound to the current thread (->tid==tid || ->tid==-2-tid).
+ * With DEBUG_TASK, the <file>:<line> from the call place are stored into the
+ * tasklet for tracing purposes.
+ *
+ * The macro accepts an optional 2nd argument that is passed as a set of flags
+ * to be set on the tasklet, among TASK_WOKEN_*, TASK_F_UEVT* etc to indicate a
+ * wakeup cause to the tasklet. When not set, the arg defaults to zero (i.e. no
+ * flag is added).
+ */
+#define tasklet_wakeup_here(tl, ...)					\
+	_tasklet_wakeup_here(tl, DEFVAL(TASK_WOKEN_OTHER, ##__VA_ARGS__), MK_CALLER(WAKEUP_TYPE_TASKLET_WAKEUP, 0, 0))
+
+static inline void _tasklet_wakeup_here(struct tasklet *tl, uint f, const struct ha_caller *caller)
+{
+	unsigned int state = _HA_ATOMIC_OR_FETCH(&tl->state, f);
+
+	do {
+		/* do nothing if someone else already added it */
+		if (state & TASK_QUEUED)
+			return;
+	} while (!_HA_ATOMIC_CAS(&tl->state, &state, state | TASK_QUEUED));
+
+	/* at this point we're the first ones to add this task to the list */
+	if (likely(caller)) {
+		caller = HA_ATOMIC_XCHG(&tl->caller, caller);
+		BUG_ON((ulong)caller & 1);
+#ifdef DEBUG_TASK
+		HA_ATOMIC_STORE(&tl->debug.prev_caller, caller);
+#endif
+	}
+
+	__tasklet_wakeup_here(tl);
+}
+
 /* schedules tasklet <tl> to run onto thread <thr> or the current thread if
  * <thr> is negative. Note that it is illegal to wakeup a foreign tasklet if
  * its tid is negative and it is illegal to self-assign a tasklet that was

src/task.c

@@ -129,6 +129,43 @@ void tasklet_kill(struct tasklet *t)
 	}
 }
 
+/* Do not call this one, please use tasklet_wakeup_here() instead, as this one
+ * is the slow path of tasklet_wakeup_here() which performs some preliminary
+ * checks and sets TASK_QUEUED before calling this one. It is only permitted to
+ * call this function for tasks/tasklets that are not bound (->tid==-1) or that
+ * are bound to the local thread (->tid==tid || ->tid==-2-tid).
+ */
+void __tasklet_wakeup_here(struct tasklet *tl)
+{
+	BUG_ON_HOT(tl->tid != -1 && tl->tid != tid && tl->tid != -2 - tid);
+
+	if (_HA_ATOMIC_LOAD(&th_ctx->flags) & TH_FL_TASK_PROFILING)
+		tl->wake_date = now_mono_time();
+
+	/* this tasklet runs on the caller thread */
+	if (tl->state & TASK_HEAVY) {
+		LIST_APPEND(&th_ctx->tasklets[TL_HEAVY], &tl->list);
+		th_ctx->tl_class_mask |= 1 << TL_HEAVY;
+	}
+	else if (tl->state & TASK_SELF_WAKING) {
+		LIST_APPEND(&th_ctx->tasklets[TL_BULK], &tl->list);
+		th_ctx->tl_class_mask |= 1 << TL_BULK;
+	}
+	else if ((struct task *)tl == th_ctx->current && !(tl->state & TASK_WOKEN_ANY)) {
+		LIST_APPEND(&th_ctx->tasklets[TL_BULK], &tl->list);
+		th_ctx->tl_class_mask |= 1 << TL_BULK;
+	}
+	else if (th_ctx->current_queue < 0) {
+		LIST_APPEND(&th_ctx->tasklets[TL_URGENT], &tl->list);
+		th_ctx->tl_class_mask |= 1 << TL_URGENT;
+	}
+	else {
+		LIST_APPEND(&th_ctx->tasklets[TL_NORMAL], &tl->list);
+		th_ctx->tl_class_mask |= 1 << TL_NORMAL;
+	}
+	_HA_ATOMIC_INC(&th_ctx->rq_total);
+}
+
 /* Do not call this one, please use tasklet_wakeup_on() instead, as this one is
  * the slow path of tasklet_wakeup_on() which performs some preliminary checks
  * and sets TASK_QUEUED before calling this one. A negative <thr> designates