Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-10-17
17:33:35 sean-k-mooney https://bugs.launchpad.net/oslo.messaging/+bug/1437955
17:33:35 openstack Launchpad bug 1437955 in oslo.messaging "RPC calls and responses do not use the mandatory flag (AMQP)" [Wishlist,Confirmed] - Assigned to Gabriele Santomaggio (gsantomaggio)
17:33:47 sean-k-mooney this kind fo explains it but i think there was more in teh review
17:36:57 sean-k-mooney while this is only the 2nd time someone has reported this in the last 4 months it is really hard to traige/identify this issue so we proably shoudl try to fix this. i guess we soudl follow up with the oslo folks and see if it ready to use
17:37:51 sean-k-mooney im surpiesed more people have not hit this to be honest
17:38:15 sean-k-mooney mayeb they have and just rebooted the node
17:41:54 melwitt yeah, should just be a matter of bumping our minimum oslo.messaging version and finding where to set the mandatory flag
17:42:15 melwitt that doesn't help for stable branches obvs. not sure if we have any options there
17:42:47 melwitt I can give it a go unless someone else wants to
17:43:54 melwitt looks like the change was released in version 9.8.0
17:46:00 melwitt thinking more, there's no way we could have this for stable branches
17:46:36 melwitt and also, how to reproduce and prove the mandatory flag fixes anything
18:05:35 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'os-consoles' API https://review.opendev.org/687907
18:05:35 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'nova-console' service, 'os-consoles' API https://review.opendev.org/687908
18:05:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909
18:12:45 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Change order of two classes https://review.opendev.org/689178
18:12:46 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Rework '_delete_server' https://review.opendev.org/689179
18:12:46 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Make '_wait_for_state_change' behave consistently https://review.opendev.org/689180
18:12:47 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Unify '_wait_until_deleted' implementations https://review.opendev.org/689181
18:12:47 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Make 'ServerTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182
18:17:51 mnaser eandersson: i have hit that exact issue many times :(
18:18:01 mnaser it really has to do with rabbitmq being unhappy when the queues come back
18:18:08 mnaser curious -- what version of it are you running?
18:31:28 eandersson 3.7.5
18:37:56 mordred stephenfin: did y'all sort out the requirements thing from earlier/
18:37:57 mordred ?
18:38:16 dansmith mordred: with the networx thing or whatever?
18:38:40 dansmith mordred: https://review.opendev.org/#/c/689152/
18:40:38 mordred dansmith: cool!
19:07:21 eandersson sean-k-mooney, mnaser > Message not delivered: NO_ROUTE (312) to queue 'compute.<compute_name>' from exchange 'nova'
19:07:24 eandersson This is the actual issue
19:07:37 eandersson It's an super odd bug because the queue is fine, and the compute can consume from it
19:07:46 eandersson but the binding (that is 100% there) simply does not work
19:09:07 eandersson if confirm-deliveries isn't enabled, which might be the problem, amqp isn't going to raise an error
19:09:49 eandersson mandatory flag isn't enough afaik
19:10:05 eandersson mandatory + confirm_deliveries is what I used
19:12:52 eandersson http://paste.openstack.org/show/784547/
19:13:55 openstackgerrit Merged openstack/nova master: Add functional recreate test for bug 1848343 https://review.opendev.org/688980
19:13:55 openstack bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem)
19:14:06 eandersson * http://paste.openstack.org/show/784548/
19:14:13 openstackgerrit Merged openstack/nova master: Add live migration recreate test for bug 1848343 https://review.opendev.org/688994
19:17:21 openstackgerrit Merged openstack/nova master: Add compute side revert allocation test for bug 1848343 https://review.opendev.org/689013
19:17:21 openstack bug 1848343 in OpenStack Compute (nova) "Reverting migration-based allocations leaks allocations if the server is deleted" [Medium,In progress] https://launchpad.net/bugs/1848343 - Assigned to Matt Riedemann (mriedem)
19:44:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete source allocations in move_allocations if target no longer exists https://review.opendev.org/689049
20:06:23 mriedem easy couple of patches to fix an upgrade issue since stein https://review.opendev.org/#/q/topic:bug/1824435+(status:open+OR+status:merged)
20:51:01 eandersson mnaser, https://github.com/rabbitmq/rabbitmq-server/pull/1884#issuecomment-464277810
20:51:04 eandersson I think this is the issue
20:52:40 eandersson https://github.com/rabbitmq/rabbitmq-server/pull/1879
20:59:57 melwitt if that's the case, then all that should be needed is upgrade rabbit to 3.7 and no need to consume the mandatory flag option in nova, iiuc
21:02:05 melwitt oh wait, according to the backscroll, need both the flag and the fix in 3.7 then sounds like
21:03:08 melwitt eandersson: can you correct me pls? ^
21:03:54 melwitt just trying to understand whether we need to do anything in nova
21:46:12 openstackgerrit Matthew Booth proposed openstack/nova master: Add new test base for libvirt functional tests https://review.opendev.org/689186
21:46:12 openstackgerrit Matthew Booth proposed openstack/nova master: Unplug VIFs as part of cleanup of networks https://review.opendev.org/663382
21:46:13 openstackgerrit Matthew Booth proposed openstack/nova master: Fix incorrect instance state after build failure https://review.opendev.org/689278
21:52:32 eandersson melwitt, so the queue can still get stuck in a bad state. It happens.
21:52:39 eandersson So having that extra layer of protection is still good.
21:53:23 eandersson (e.g. network partitions etc)
21:53:40 melwitt eandersson: sorry, what I mean is, what do we need to do in nova, if anything? pass the mandatory flag? pass the mandatory flag and the confirm_deliveries flag? or is nothing needed?
21:53:56 eandersson I think this probably has to be done on the oslo.messaging side
21:54:17 melwitt I wasn't sure whether your findings meant a change in the plan for
21:54:21 melwitt nova
21:54:23 eandersson but I still feel like a vm shouldn't be able to stay in a state like that indefinitely
21:55:24 eandersson maybe that could be changed to an rpc call instead?
21:55:51 melwitt no, it shouldn't. sorry, somehow what I'm saying is not coming across. you posted links about rabbit bugs earlier, so I wasn't sure if you were saying it's a rabbit bug and there's nothing we can do to help in nova
21:57:29 melwitt this is the change they made in oslo.messaging to allow people (like nova) to pass flags https://review.opendev.org/660373
21:58:40 eandersson Yea - if they is exposed I would add mandatory to these calls in nova.
21:59:16 melwitt ok, cool
22:02:11 eandersson It will help protect against a bunch of potential RabbitMQ issues
22:03:47 melwitt thanks for clarifying that. I'll find how to add the flag on the nova side
22:27:08 eandersson melwitt, some information on mandatory here https://www.rabbitmq.com/reliability.html#routing
22:50:02 melwitt eandersson: thx
23:17:48 openstackgerrit Merged openstack/nova stable/stein: Stop sending bad values from libosinfo to libvirt https://review.opendev.org/688067
23:17:55 openstackgerrit Merged openstack/nova stable/queens: Add functional recreate test for regression bug 1825537 https://review.opendev.org/675355
23:17:55 openstack bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem)
#openstack-nova - 2019-10-18
00:18:12 openstackgerrit Merged openstack/nova stable/queens: Fix rebuild of baremetal instance when vm_state is ERROR https://review.opendev.org/680873
04:41:37 openstackgerrit Merged openstack/nova master: Add image caching API for aggregates https://review.opendev.org/687140
05:06:31 openstackgerrit Merged openstack/nova master: Fix up some feedback on image precache support https://review.opendev.org/688172
07:05:58 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] To create single NUMA node instance in function '_get_numa_topology_auto' https://review.opendev.org/688932
07:09:30 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] Assign and track instance pinning cpu through 'cpu_pinning' field https://review.opendev.org/688933
07:11:15 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] Add a new instance CPU allocation policy: mixed https://review.opendev.org/688934
07:11:17 openstackgerrit Huachang Wang proposed openstack/nova master: [WIP] virt/libvirt: Get host pin cpuset according instance cpu_pinning https://review.opendev.org/688935
07:28:19 bauzas morning fellows
07:41:37 openstackgerrit Brin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy https://review.opendev.org/674038
07:43:28 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies https://review.opendev.org/669181
07:43:42 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-agents policies https://review.opendev.org/669578
07:44:27 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-services https://review.opendev.org/645427
07:44:55 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class https://review.opendev.org/645452
07:59:00 openstackgerrit Brin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy https://review.opendev.org/674038
07:59:01 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies https://review.opendev.org/669181
07:59:01 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-services https://review.opendev.org/645427
07:59:02 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class https://review.opendev.org/645452
07:59:02 openstackgerrit Brin Zhang proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests https://review.opendev.org/676670
08:01:08 openstackgerrit Brin Zhang proposed openstack/nova master: Pass RequestContext to oslo_policy https://review.opendev.org/674038
08:03:27 openstackgerrit Brin Zhang proposed openstack/nova master: Add test coverage of existing os-services policies https://review.opendev.org/669181
08:03:34 openstackgerrit Brin Zhang proposed openstack/nova master: Introduce scope_types in os-services https://review.opendev.org/645427
08:04:23 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
08:07:23 openstackgerrit Brin Zhang proposed openstack/nova master: Add new default roles and mapping in policy base class https://review.opendev.org/645452
08:30:56 openstackgerrit Brin Zhang proposed openstack/nova master: Add policy deprecation fixture and Suppress warnings in tests https://review.opendev.org/676670
09:14:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Refactor rebuild_instance https://review.opendev.org/688419

Earlier   Later