Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
21:25:42 eandersson I know there is a patch to add mandatory to nova
21:25:52 eandersson the mandatory flag when publishing messages
21:26:23 eandersson Since if a message is lost you are stuck in buidling forever
21:26:29 eandersson with no error
21:27:01 mnaser my favorite :)
21:33:32 eandersson Gonna wipe the rmq db and see if it goes away
21:34:05 eandersson It's just odd that our synth tests are not hitting this
21:34:22 eandersson We create 300 VMs per day and none end up in this state
21:49:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add node owner pre-filter https://review.opendev.org/697187
21:54:31 mriedem dansmith: this is probably a blast from the past for you https://review.opendev.org/#/c/697122/
21:54:33 mriedem mitaka era
22:09:58 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162
22:11:32 dansmith mriedem_away: I don't remember it, but I totes believe I did something awesome that someone else broke
23:00:51 sean-k-mooney eandersson: the patch for the mandatory flag i think is more or less stalled i have not seen anything happen on the fron in a while
23:01:37 eandersson Awh that is too bad
23:02:07 sean-k-mooney it looks like all the oslo.messaging fixes are done https://review.opendev.org/#/q/status:merged+project:openstack/oslo.messaging+branch:master+topic:bp/transport-options
23:02:25 sean-k-mooney but i dont think anyone has gotten around to using that in nova yet
23:02:43 eandersson Yea - and would probably not be easy to backport.
23:02:45 eandersson Or even possible.
23:03:33 sean-k-mooney well we need to do too things. frist start using it in master and second we need to figure out what the correct action is to take if we get an exception indicating it could not be delivered
23:03:44 eandersson I think I am going to add a log to nova just before it sends that notification to the computes.
23:05:01 sean-k-mooney for example if we could not deliver a message to a compute queue should we disabel that compute node so we dont try to schdule to it agian?
23:05:09 sean-k-mooney shoudl we jsut retry?
23:06:30 sean-k-mooney i actully have no idea what the best path forward would be in that case but we kind of need to figure that out before we can fix the issue
23:06:48 sean-k-mooney we now have the feature that allows use to adress it however
23:11:24 sean-k-mooney melwitt: did you ever have time to talk to the oslo folks on how we could use this https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options and specificly the mandaroty flag so we can adress the nova aspect of https://bugs.launchpad.net/oslo.messaging/+bug/1661510
23:11:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
23:12:05 sean-k-mooney i know i havent but it looks like all the patchs are in place on the oslo side
23:28:23 melwitt sean-k-mooney: I haven't talked to them bc like you said, their side is done. it's up to us (me) to figure out how to use it on the nova side and I haven't gotten a chance to dig in to it yet
23:28:54 melwitt I don't yet know where/how to pass it in the nova code
23:29:51 sean-k-mooney ya i just asked on the bug if they could provdie an example
23:30:51 sean-k-mooney i know its something we shoudl be setting when we do a topic send which is i guess a call to a compute node primarly
23:31:11 sean-k-mooney but i dont know what we should do if we are not able to deliver it to the message queue
23:40:40 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
23:44:56 gmann johnthetubaguy: this should be ready now. I was checking why project-admin and legacy admin can access the new system reader rule. which was correct because of old defaults(admin_api) are deprecated and still work. This make sure any old token will keep working for changed defaults also.
23:45:29 gmann johnthetubaguy: once you are ok with this then it will complete the first set of change - https://review.opendev.org/648480
23:51:23 openstackgerrit sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957
23:51:24 openstackgerrit sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861
23:56:08 efried https://bugs.launchpad.net/neutron/+bug/1855015
23:56:09 openstack Launchpad bug 1855015 in OpenStack Compute (nova) "Intermittent fails since 11/23 with "Multiple possible networks found, use a Network ID to be more specific."" [Undecided,New]
23:56:57 sean-k-mooney has that reappared
23:57:37 sean-k-mooney that is a tempset bug and or we have not confgiured tempet correctly
23:58:32 eandersson If we can’t deliver to the queue I think we should just set to error or move to the next available compute
23:58:52 sean-k-mooney eandersson: set what to error?
23:58:57 eandersson The instance
23:59:12 sean-k-mooney that would not be correct in all cases
23:59:54 sean-k-mooney if the call was to say the instace diagnostics endpoint
#openstack-nova - 2019-12-04
00:00:13 sean-k-mooney we shoudl not put the instance in error state because we were not able to get the diagnositcs right
00:00:17 eandersson sure this would only be for the scheduling event
00:00:29 eandersson since mandatory is per message not per queue
00:01:00 sean-k-mooney its per type of rpc call we do
00:01:15 sean-k-mooney e.g. you set it in your argmuent to send
00:02:59 sean-k-mooney if we pass mandaroy on all topic sends then we woudl ahve to deally with it constently for ever api action that needs to call the compute node
00:03:29 sean-k-mooney if we dont set it gloably for all topic sends we need to consdider how to handel the failure specific for each action
00:06:30 sean-k-mooney eandersson: in the specifc case of server create. if the downcall form the condutor to the compute failed to send you would like us to retry the next compute node correct.
00:07:04 sean-k-mooney if we set the instance to error at that point we would not retry
00:09:27 sean-k-mooney if we did retry i think we would still want to incremetn the failed boot counter for the compute node so the BuildFailureWeigher can avoid that host and log an error at a minium
00:09:59 eandersson Sorry stepping away again. Btw we got a ton of open positions if anyone wants to come and help me work on this stuff. :p
00:10:38 sean-k-mooney no worries im about to call it a night anyway
01:45:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688
01:45:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
02:22:14 openstackgerrit Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963
02:27:12 eandersson Restart rmq and rebuilt it. Lets see if the issue goes away.
02:29:04 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969
03:04:33 openstackgerrit Brin Zhang proposed openstack/nova master: PoC: Support re-configure the delete_on_termination in server https://review.opendev.org/693828
03:24:17 eandersson sean-k-mooney yea I would prefer it to retry and write an error to the logs, but probably cleaner and easier to implement it to stop and put the instance into an ERROR'd state instead.
03:25:03 eandersson I am gonna see if I can set up something like Terraform to better handle instances stuck in BUILDING
03:25:28 eandersson I think out of the box it gets stuck in a weird state, but I am sure there is a way to have Terraform handle issues like this more gracefully.
03:59:10 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
05:05:47 openstackgerrit jichenjc proposed openstack/nova master: downgrade when host does not support capabilities https://review.opendev.org/697228
05:47:26 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336
06:46:55 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717
07:37:35 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
07:56:29 openstackgerrit Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856
09:08:52 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242
09:08:52 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
09:08:53 openstackgerrit Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707
09:08:53 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
09:08:54 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into VM's domain XML. https://review.opendev.org/631245
09:08:54 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
09:08:55 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
09:57:38 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove remaining nova-network quotas https://review.opendev.org/697152
09:57:38 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'FloatingIP' object https://review.opendev.org/697153
09:57:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'Network' object https://review.opendev.org/697154
09:57:39 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove use of legacy 'SecurityGroup' object https://review.opendev.org/697155
09:57:40 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Remove unused nova-network objects https://review.opendev.org/697156
09:57:40 openstackgerrit Stephen Finucane proposed openstack/nova master: nova-net: Follow-up for I12a96ea659ed402cc4d1bd52a50e2e16042b6372 https://review.opendev.org/697252
09:59:10 stephenfin bauzas: Care to get some of that nova-net removal goodness in ya this morning? https://review.opendev.org/#/c/696503/
09:59:21 bauzas stephenfin: I could
10:36:49 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260
10:37:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261
11:34:58 openstackgerrit Thierry Carrez proposed openstack/nova master: Remove unused rootwrap filters https://review.opendev.org/697134
12:34:38 gibi gmann: if you have time, this tempest fix would help with some of the tempest failures in CI https://review.opendev.org/#/c/682964
12:42:59 efried o/ gibi stephenfin, lend me your ears for a moment?
12:43:38 gibi efried: I'm here but expect me to dissapeare randomly as I'm at the doctor's waiting room
12:44:25 lyarwood efried: I also think stephenfin is AFK at lunch at the moment btw
12:46:16 efried okay, thanks, I'll just leave this here then:
12:46:16 efried I'm working on this virtual TPM thing, trying to get the spec approved.
12:46:16 efried https://review.opendev.org/#/c/686804/
12:46:16 efried Dan helped me get it to this point, but doesn't like some of the remaining oddness around snapshot-related operations.

Earlier   Later