vListInsert for( pxIterator… annoying program hang

Hello, I use freeRTOS v8.2.0 on a Cortex M4 (NXP LPC4078) and am getting a program crash in list.c, vListInsert() – function. It’s hanging in the for-loop where the 4 possibilities are described above. I’m searching for days in the freeRTOS forum, google, etc. but problem still exists and I didn’t get a solution yet. In debugger it seems that it occurs on calling a xTaskNotifyWait() – command which calls in the middle taskEXITCRITICAL(); and taskENTERCRITICAL(); where it doesn’t continue. configASSERT() is defined and already helped me at some other problems. What about the 4 possibilites?
        1) Stack overflow -
           see http://www.freertos.org/Stacks-and-stack-overflow-checking.html
I set #define configCHECKFORSTACK_OVERFLOW 1 and checked for vApplicationStackOverflowHook. It’s never called and never stays there in the endless loop. In addition I checked the running tasks with its stack usage before calling xTaskNotifyWait(): ~~~ File Transfer task R 2 268 19 AnyBus R 0 150 3 IDLE R 0 93 12 Tmr Svc B 3 186 13 NFC Tag B 0 262 6 console B 7 264 11 LED Control B 2 195 1 Event task B 2 178 15 ADC Control B 2 218 7 ext. ADC B 2 90 5 Eeprom Control B 2 207 8 abccCmdHandler B 4 583 10 LSB Driver S 0 175 2 ~~~ “File Transfer task” is the running task, where vListInsert() – crash occurs.
        2) Incorrect interrupt priority assignment, especially on Cortex-M
           parts where numerically high priority values denote low actual
           interrupt priorities, which can seem counter intuitive.  See
           http://www.freertos.org/RTOS-Cortex-M3-M4.html and the definition
           of configMAX_SYSCALL_INTERRUPT_PRIORITY on
           http://www.freertos.org/a00110.html
I checked both websites, but unfortunately I do not completely understand Cortex M4 interrupt handling (it’s a bit complex:-)). In FreeRTOSConfig.h the following values are set:
#define configLIBRARY_LOWEST_INTERRUPT_PRIORITY         0x3f
#define configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY    5
#define configPRIO_BITS       __NVIC_PRIO_BITS   // which is 5
I already tried values 0-15 for configLIBRARYMAXSYSCALLINTERRUPTPRIORITY, that doesn’t change anything. But since it seems to crash in taskEXIT_CRITICAL() I would assume that it could occur in realtion to interrupt handling?
        3) Calling an API function from within a critical section or when
           the scheduler is suspended, or calling an API function that does
           not end in "FromISR" from an interrupt.
As I wrote it seems to occur in xTaskNotifyWait() – command and there in the middle at taskEXIT_CRITICAL();. The function isn’t modified in any way.
        4) Using a queue or semaphore before it has been initialised or
           before the scheduler has been started (are interrupts firing
           before vTaskStartScheduler() has been called?).
Since program is running fine before, all tasks are created, calling vTaskStartScheduler() is long in the past. The vListInsert() – crash doesn’t occur in initialization before scheduler started. It occurs on any actions when program run’s fine. Do you need any additional information from me? It would be really great if you have any hints what could cause the problem or which things I should check. Project is a bit in hurry and it’s annoying that I can’t continue its development due to this crash. Thank you in advance! Regards, Daniel

vListInsert for( pxIterator… annoying program hang

That’s the same symptom I have: https://sourceforge.net/p/freertos/discussion/382005/thread/04c22d6e9a/?limit=25#843e/f6dc I still have no clue what’s causing it but you’re not alone, if that makes you feel any better. :-/ If anyone gives suggestions for things to check I’ll also check on my end, maybe find a clue that will help sort this out.

vListInsert for( pxIterator… annoying program hang

Thanks for the info. It seems I don’t need to update to FreeRTOS 10. Maybe it has something to do with your stop mode etc. but my program is running in stable idle mode. Richard Barry wrote in your post that
it’s often a symptom of incorrect interrupt priority configurations
I read that the past days so many times but what can I do for it? What exactly could be wrong in the priority, what is not properly configured? This is my interrupt table of cmsis40xx.h (unfortunately the textbox here is not wide enough…): ~~~ typedef enum { /* ————————- Cortex-M4 Processor Exceptions Numbers —————————– */ ResetIRQn = -15, /!< 1 Reset Vector, invoked on Power up and warm reset */ NonMaskableInt_IRQn = -14, /!< 2 Non maskable Interrupt, cannot be stopped or preempted / HardFault_IRQn = -13, /!< 3 Hard Fault, all classes of Fault / MemoryManagement_IRQn = -12, /!< 4 Memory Management, MPU mismatch, including Access Violation and No Match / BusFault_IRQn = -11, /!< 5 Bus Fault, Pre-Fetch-, Memory Access Fault, other address/memory related Fault / UsageFault_IRQn = -10, /!< 6 Usage Fault, i.e. Undef Instruction, Illegal State Transition / SVCall_IRQn = -5, /!< 11 System Service Call via SVC instruction / DebugMonitor_IRQn = -4, /!< 12 CDebug Monitor / PendSV_IRQn = -2, /!< 14 Pendable request for system service / SysTick_IRQn = -1, /!< 15 System Tick Interrupt */
/* ---------------------------  LPC40xx Specific Interrupt Numbers  ------------------------------- */
WDT_IRQn                      = 0,          /*!< Watchdog Timer Interrupt                         */
TIMER0_IRQn                   = 1,          /*!< Timer0 Interrupt                                 */
TIMER1_IRQn                   = 2,          /*!< Timer1 Interrupt                                 */
TIMER2_IRQn                   = 3,          /*!< Timer2 Interrupt                                 */
TIMER3_IRQn                   = 4,          /*!< Timer3 Interrupt                                 */
UART0_IRQn                    = 5,          /*!< UART0 Interrupt                                  */
UART_IRQn                     = UART0_IRQn, /*!< Alias for UART0 Interrupt                        */
UART1_IRQn                    = 6,          /*!< UART1 Interrupt                                  */
UART2_IRQn                    = 7,          /*!< UART2 Interrupt                                  */
UART3_IRQn                    = 8,          /*!< UART3 Interrupt                                  */
PWM1_IRQn                     = 9,          /*!< PWM1 Interrupt                                   */
I2C0_IRQn                     = 10,         /*!< I2C0 Interrupt                                   */
I2C_IRQn                      = I2C0_IRQn,  /*!< Alias for I2C0 Interrupt                         */
I2C1_IRQn                     = 11,         /*!< I2C1 Interrupt                                   */
I2C2_IRQn                     = 12,         /*!< I2C2 Interrupt                                   */
Reserved0_IRQn                = 13,         /*!< Reserved                                         */
SSP0_IRQn                     = 14,         /*!< SSP0 Interrupt                                   */
SSP_IRQn                      = SSP0_IRQn,  /*!< Alias for SSP0 Interrupt                         */
SSP1_IRQn                     = 15,         /*!< SSP1 Interrupt                                   */
PLL0_IRQn                     = 16,         /*!< PLL0 Lock (Main PLL) Interrupt                   */
RTC_IRQn                      = 17,         /*!< Real Time Clock Interrupt                        */
EINT0_IRQn                    = 18,         /*!< External Interrupt 0 Interrupt                   */
EINT1_IRQn                    = 19,         /*!< External Interrupt 1 Interrupt                   */
EINT2_IRQn                    = 20,         /*!< External Interrupt 2 Interrupt                   */
EINT3_IRQn                    = 21,         /*!< External Interrupt 3 Interrupt                   */
ADC_IRQn                      = 22,         /*!< A/D Converter Interrupt                          */
BOD_IRQn                      = 23,         /*!< Brown-Out Detect Interrupt                       */
USB_IRQn                      = 24,         /*!< USB Interrupt                                    */
CAN_IRQn                      = 25,         /*!< CAN Interrupt                                    */
DMA_IRQn                      = 26,         /*!< General Purpose DMA Interrupt                    */
I2S_IRQn                      = 27,         /*!< I2S Interrupt                                    */
ETHERNET_IRQn                 = 28,         /*!< Ethernet Interrupt                               */
SDC_IRQn                      = 29,         /*!< SD/MMC card I/F Interrupt                        */
MCPWM_IRQn                    = 30,         /*!< Motor Control PWM Interrupt                      */
QEI_IRQn                      = 31,         /*!< Quadrature Encoder Interface Interrupt           */
PLL1_IRQn                     = 32,         /*!< PLL1 Lock (USB PLL) Interrupt                    */
USBActivity_IRQn              = 33,         /*!< USB Activity interrupt                           */
CANActivity_IRQn              = 34,         /*!< CAN Activity interrupt                           */
UART4_IRQn                    = 35,         /*!< UART4 Interrupt                                  */
SSP2_IRQn                     = 36,         /*!< SSP2 Interrupt                                   */
LCD_IRQn                      = 37,         /*!< LCD Interrupt                                    */
GPIO_IRQn                     = 38,         /*!< GPIO Interrupt                                   */
PWM0_IRQn                     = 39,         /*!< PWM0 Interrupt                                   */
EEPROM_IRQn                   = 40,         /*!< EEPROM Interrupt                               */
} LPC40XXIRQnType; ~~~

vListInsert for( pxIterator… annoying program hang

One thing I see that seems wrong is that your comments say that NVICPRIOBITS is 5, which means you have priorities 0-31 (0x1F), but configLIBRARYLOWESTINTERRUPTPRIORITY is 0x3F (63) which is 6 bits. Cortex interrupts aren’t that difficult, they just have a couple of things that can cause confusion. First, low priority is big numbers, all ones is the lowest priority and 0 is the highest (and normally above the configLIBRARYMAXSYSCALLINTERRUPTPRIORITY limit and the default for an interrupt) Second, they normally aren’t 8 bits bit, and you need to look carefully at the functions you call to see if they want the priority in the bottom bits or the top bits of the byte. The actual hardware wants the interrupt priority in the top bits of the byte, and may use some of the lower bits for a ‘sub-priority’ that just choses which interrupt to prefer, but doesn’t create interrupt nesting. The table you posted above actually has almost nothing to do about interrupt priority, but are the vector numbers for the various interrupts (used as a parameter to the various function calls that set priority/enable/disable/clear the various interrupts. Generally every interupt you use should have in your code (or a library function you call for it) a call to something like NVICSETPRIORITY(vector number, priority) , and then NVICENABLEINT(vector number) (I am going by memory on the name of the functions at the moment).

vListInsert for( pxIterator… annoying program hang

Hi Richard, I already changed configLIBRARYLOWESTINTERRUPT_PRIORITY to 0x1F but it doesn’t help. After some investigation I checked NVIC->IP Block in memory as well as SCB->SHP Block. There I should see the current priorities? There are alot of calls with NVICSETPRIORITY() and NVICENABLEINT(), it’s hard to check them all separately. In memory I see at NVIC->IP adress the first 40 bytes as (I hope to interpret the indexes correctly): ~~~ 00 WDTIRQn
00 TIMER0
IRQn 00 TIMER1IRQn 00 TIMER2IRQn 00 TIMER3IRQn 08 UART0IRQn 00 UART1IRQn 00 UART2IRQn 00 UART3IRQn 00 PWM1IRQn 00 I2C0IRQn 00 I2C1IRQn
00 I2C2IRQn
00 Reserved0
IRQn 00 SSP0IRQn
00 SSP1
IRQn
00 PLL0IRQn
00 RTC
IRQn
00 EINT0IRQn
00 EINT1
IRQn
00 EINT2IRQn
00 EINT3
IRQn
28 ADCIRQn
00 BOD
IRQn
00 USBIRQn
28 CAN
IRQn
28 DMAIRQn
00 I2S
IRQn
00 ETHERNETIRQn
00 SDC
IRQn
00 MCPWMIRQn
00 QEI
IRQn
00 PLL1IRQn
00 USBActivity
IRQn 00 CANActivityIRQn 00 UART4IRQn
00 SSP2IRQn
00 LCD
IRQn
00 GPIOIRQn
00 PWM0
IRQn
00 EEPROM_IRQn ~~~ The function which sets the interrupts looks like this: ~~~ __STATICINLINE void NVICSetPriority(IRQnType IRQn, uint32t priority) { if(IRQn < 0) { SCB->SHP[((uint32t)(IRQn) & 0xF)-4] = ((priority << (8 – __NVICPRIOBITS)) & 0xff); } /* set Priority for Cortex-M System Interrupts */ else { NVIC->IP[(uint32t)(IRQn)] = ((priority << (8 – __NVICPRIOBITS)) & 0xff); } /* set Priority for device specific Interrupts */ } ~~~ I interpret ADCIRQn, CANIRQn and DMAIRQn as priority 5 and UART0IRQn as 1. SCB->SHP block: ~~~ 00 MemoryManagementIRQn 00 BusFaultIRQn 00 UsageFaultIRQn 00 00 00 00 00 SVCallIRQn 00 DebugMonitorIRQn 00 F8 PendSVIRQn F8 SysTick_IRQn ~~~ SysTickIRQn is set by this call: ~~~ NVICSetPriority (SysTick_IRQn, (1<<__NVIC_PRIO_BITS) – 1); /* set Priority for Systick Interrupt */ ~~~ Is that lowest priority? I can’t find any call for SVCall_IRQn, so it isn’t set? Is “0” highest priority?

vListInsert for( pxIterator… annoying program hang

~~~ LPC40XXIRQnType ~~~ These are not interrupt priorities, just the number of each interrupt in your system. The latest version of the FreeRTOS code, with configASSERT() defined, will catch nearly all, if not actually all, misconfigurations.

vListInsert for( pxIterator… annoying program hang

The mentioned NVICSetPriority (SysTickIRQn… isn’t used but configLIBRARYLOWESTINTERRUPTPRIORITY of FreeRTOSConfig.h. I changed PendSVIRQn and SysTick_IRQn to highest priority 0x08 (5 bits left aligned means it’s priority 1) but then I get a HardFaultHandler call during initialization. Beginning with prio 5 idle runs again fine. Are there any other things I can check to be sure interrupts are set correctly? Do you need any more information?

vListInsert for( pxIterator… annoying program hang

The systick and pendsv priorities should be the lowest (especially pendsv). As I recall they are set in the kernel code itself. It is possible to run systick with a higher priority, and some ST drivers require it to be the highest as they poll the tick count in lower priority interrupt (?).

vListInsert for( pxIterator… annoying program hang

It’s not an ST µC but a NXP LPC4078, therefore I use the lpcchip40xx library. As far as I know lowest priority “31” (=0x1F shifted <<3 = 0xF8) should be set for both. Is there a way to find out why it hangs in for-loop of vListInsert()? I checked the parameters of for loops but seems very complex. Step out in debugger doesn’t really work, it jumps out to higher functions but they do not belong to it… This annoying hang blocks my further project work. Is there something more to check?

vListInsert for( pxIterator… annoying program hang

Generally malfunctioning vListInsert (and related) is NOT an issue at the immediate call site, but some problem earlier that has corrupted the lists, so they don’t meet the expected form. One big cause of this is the incorrect setting of interrupt priority, so that the critical sections don’t work. A second be source is corrupting memory by exceed array bounds or stack overflow. Arm processors using the NVIC are particually prone to interrupt priority issues as the default of 0 for priority, which means absolutly highest, can cause issues, and the shifted priority register (and needing to determine if a given call want the priority in the bottom or top n bits of the byte) gives more opertunity for issues. Later version of FreeRTOS added a LOT more checks for interupt priority using configASSERT(), which can be a big help.

vListInsert for( pxIterator… annoying program hang

Hi Richard, I think to understand it. Now should I set every interrupt to at least configLIBRARYMAXSYSCALLINTERRUPTPRIORITY (which is 5)? The vector table in crstartuplpc407x_8x.x contains a lot of interrupts and basically not all of them are activated or used. Should I check which are enabled and set the priority of them to 5 or higher? What about the system interrupts at the beginning of following list, are there interrupts which may be < 5 or even 0? ~~~ extern void (* const gpfnVectors[])(void); attribute ((section(“.isrvector”))) void (* const gpfnVectors[])(void) = { // Core Level – CM4 &vStackTop, // The initial stack pointer ResetISR, // The reset handler NMIHandler, // The NMI handler HardFaultHandler, // The hard fault handler MemManageHandler, // The MPU fault handler BusFaultHandler, // The bus fault handler UsageFaultHandler, // The usage fault handler 0, // Reserved 0, // Reserved 0, // Reserved 0, // Reserved SVCHandler, // SVCall handler DebugMonHandler, // Debug monitor handler 0, // Reserved PendSVHandler, // The PendSV handler SysTick_Handler, // The SysTick handler
// Chip Level - LPC40xx
WDT_IRQHandler,                     // 16, 0x40 - WDT
TIMER0_IRQHandler,                  // 17, 0x44 - TIMER0
TIMER1_IRQHandler,                  // 18, 0x48 - TIMER1
TIMER2_IRQHandler,                  // 19, 0x4c - TIMER2
TIMER3_IRQHandler,                  // 20, 0x50 - TIMER3
UART0_IRQHandler,                   // 21, 0x54 - UART0
UART1_IRQHandler,                   // 22, 0x58 - UART1
UART2_IRQHandler,                   // 23, 0x5c - UART2
UART3_IRQHandler,                   // 24, 0x60 - UART3
PWM1_IRQHandler,                    // 25, 0x64 - PWM1
I2C0_IRQHandler,                    // 26, 0x68 - I2C0
I2C1_IRQHandler,                    // 27, 0x6c - I2C1
I2C2_IRQHandler,                    // 28, 0x70 - I2C2
IntDefaultHandler,                  // 29, Not used
SSP0_IRQHandler,                    // 30, 0x78 - SSP0
SSP1_IRQHandler,                    // 31, 0x7c - SSP1
PLL0_IRQHandler,                    // 32, 0x80 - PLL0 (Main PLL)
RTC_IRQHandler,                     // 33, 0x84 - RTC
EINT0_IRQHandler,                   // 34, 0x88 - EINT0
EINT1_IRQHandler,                   // 35, 0x8c - EINT1
EINT2_IRQHandler,                   // 36, 0x90 - EINT2
EINT3_IRQHandler,                   // 37, 0x94 - EINT3
ADC_IRQHandler,                     // 38, 0x98 - ADC
BOD_IRQHandler,                     // 39, 0x9c - BOD
USB_IRQHandler,                     // 40, 0xA0 - USB
CAN_IRQHandler,                     // 41, 0xa4 - CAN
DMA_IRQHandler,                     // 42, 0xa8 - GP DMA
I2S_IRQHandler,                     // 43, 0xac - I2S

if defined (__USE_LPCOPEN)

ETH_IRQHandler,                     // 44, 0xb0 - Ethernet
SDIO_IRQHandler,                    // 45, 0xb4 - SD/MMC card I/F

else

ENET_IRQHandler,                    // 44, 0xb0 - Ethernet
MCI_IRQHandler,                     // 45, 0xb4 - SD/MMC card I/F

endif

MCPWM_IRQHandler,                   // 46, 0xb8 - Motor Control PWM
QEI_IRQHandler,                     // 47, 0xbc - Quadrature Encoder
PLL1_IRQHandler,                    // 48, 0xc0 - PLL1 (USB PLL)
USBActivity_IRQHandler,             // 49, 0xc4 - USB Activity interrupt to wakeup
CANActivity_IRQHandler,             // 50, 0xc8 - CAN Activity interrupt to wakeup
UART4_IRQHandler,                   // 51, 0xcc - UART4
SSP2_IRQHandler,                    // 52, 0xd0 - SSP2
LCD_IRQHandler,                     // 53, 0xd4 - LCD
GPIO_IRQHandler,                    // 54, 0xd8 - GPIO
PWM0_IRQHandler,                    // 55, 0xdc - PWM0
EEPROM_IRQHandler,                  // 56, 0xe0 - EEPROM
}; ~~~

vListInsert for( pxIterator… annoying program hang

I think to understand it. Now should I set every interrupt to at least configLIBRARYMAXSYSCALLINTERRUPTPRIORITY (which is 5)?
They will default to that anyway, so no point in setting them to that. You need to set the priorities to be correct for your application. If that means they are all the same, then it is probably more appropriate to set them all low rather than high.
The vector table in crstartuplpc407x_8x.x contains a lot of interrupts and basically not all of them are activated or used. Should I check which are enabled
They will be disabled by default, so no need to check if they are enabled. If you enable one, you will know its enabled.

vListInsert for( pxIterator… annoying program hang

I found out that I2C0IRQHandler and GPIOIRQHandler were enabled, too, but with Prio 0. I changed it to 5 (0x28) but unfortunately…. doesn’t help. These two ISR functions don’t call any FreeRTOS API functions maybe therefore it isn’t ciritical. Concerning stack usage, currently I set configTOTALHEAPSIZE to 40kb and tried to increase usStackDepth of the active tasks. I got following HighWatermarks from vTaskList(): ~~~ File Transfer task R 2 396 17 AnyBus R 0 284 3 IDLE R 0 221 12 NFC Tag B 0 260 6 abccCmdHandler B 4 583 10 console B 7 264 11 LED Control B 2 195 1 Event task B 2 182 15 ADC Control B 2 216 7 ext. ADC B 2 350 5 Eeprom Control B 2 207 8 Tmr Svc B 3 186 13 LSB Driver S 0 303 2 ~~~ Shouldn’t that be enough?

vListInsert for( pxIterator… annoying program hang

Little more information: I get the error only if my #define ABCCCFGDEBUG_MESSAGING is enabled which activates debug messaging. During xTaskNotifyWait another tasks continues to run in background to wait for an answer on SPI lane. This task some time calls the “debug message” function which leads to a xQueueSend – call to trigger the UART task to send the debug message. But inside xQueueSend (= xQueueGenericSend) several times vTaskPlaceOnEventList -> vListInsert will be called. At the third call it hangs at for loop. The first parameter pxEventList of vTaskPlaceOnEventList at this time contains: ~~~ pxQueue Queuet * const 0x100078a8 <ucHeap+22460>
pcHead int8
t * 0x100078fc <ucHeap+22544> “02f”
*pcHead int8_t 2 ‘02’
pcTail int8_t * 0x1000799c <ucHeap+22704> “”
*pcTail int8_t 0 ‘’
pcWriteTo int8_t * 0x1000793c <ucHeap+22608> “0226”
*pcWriteTo int8_t 2 ‘02’
u union {…} {…}
pcReadFrom int8_t * 0x1000792c <ucHeap+22592> “02f”
uxRecursiveCallCount UBaseType_t 268466476
xTasksWaitingToSend List_t {…}
uxNumberOfItems UBaseType_t 2
pxIndex ListItem_t * 0x100078c0 <ucHeap+22484>
xListEnd MiniListItem_t {…}
xTasksWaitingToReceive List_t {…}
uxNumberOfItems UBaseType_t 0
pxIndex ListItem_t * 0x100078d4 <ucHeap+22504>
xListEnd MiniListItem_t {…}
uxMessagesWaiting volatile UBaseType_t 10
uxLength UBaseType_t 10
uxItemSize UBaseType_t 16
xRxLock volatile BaseType_t 0
xTxLock volatile BaseType_t 0
uxQueueNumber UBaseType_t 0
ucQueueType uint8_t 0 ‘’
~~~ Is there any detail to find out? Or does it more seem that some time before the variable pxQueue was (partly) overwritten due to any stack overflow or wrong interrupt handling and now for-loop in vListInsert doesn’t continue?

vListInsert for( pxIterator… annoying program hang

We solved it! A further stack in our software library often disables and enables all interrupts because there are some critical sections. In one of these critical sections the FreeRTOS function xQueueSend for another UART task is called where in a special case the queue is full and it waits …. but since all interrupts are disable, SysTick – interrupt and therefore the whole FreeRTOS isn’t running any more so that UART task cannot process the queue by xQueueReceive. This is an endless loop (deadlock). One possibility could be to use xSemaphoreTake and Give instead of interrupt disables. With semaphores it’s possible to block critical sections so that they aren’t interrupted by other functions while interrupts are still running and UART task processes the queue. Thanks for all of your tipps!