Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
18:37:29 mriedem but soon, very soon....muwahhaaha
18:40:02 openstackgerrit Matt Riedemann proposed openstack/nova master: Add cross-cell resize tests for _poll_unconfirmed_resizes https://review.opendev.org/698322
18:40:03 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304
18:41:25 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: debug cross-cell resize https://review.opendev.org/698304
18:44:19 mriedem efried: stephenfin's changes look fine to me
18:45:00 efried +A
18:45:26 mriedem if someone can push through https://review.opendev.org/#/c/696509/ it's the current bottom of the nova-net removal series; it looks like the rest of the series after that is now in merge conflict so the whole thing has to be rebased.
18:52:46 efried mriedem: I'm gonna try to hit that today, but it keeps getting pushed down my stack :(
18:53:39 mriedem ack it's pretty mechanical so anyone should be able to hit it
19:02:40 melwitt TheJulia: do you know whether cpu_arch is supposed to be required from an ironic pov? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L103-L106 is it valid for a deployment *not* to specify cpu_arch? (for example, in a single arch environment) context is this bug https://bugzilla.redhat.com/show_bug.cgi?id=1688838
19:02:40 openstack bugzilla.redhat.com bug 1688838 in openstack-nova "Ironic should not treat cpu_arch as mandatory" [Medium,New] - Assigned to mwitt
19:06:12 sean-k-mooney gibi: by the way i assume attaching a port with a resocue request to a existing instance is still supported correct. it was declared out of scope in stien and it was not mentioned as adressed in train. is that on your ussuri todo list?
19:08:19 sean-k-mooney i assume we would have updated https://github.com/openstack/nova/blob/master/releasenotes/notes/reject-interface-attach-with-port-resource-request-17473ddc5a989a2a.yaml if it was supported or added another release note
19:18:41 melwitt jroll: ^ maybe you might know (ironic driver question from me)
19:31:46 mriedem sean-k-mooney: not supported - the request has to be validated with placement on attach and that isn't done
19:32:11 sean-k-mooney ya that is what i understood too
19:32:27 sean-k-mooney im doing a downstream docs review and wanted to make sure that was called out
19:33:13 mnaser eandersson: is this similar to what you've been running into? https://bugs.launchpad.net/nova/+bug/1835637
19:33:13 openstack Launchpad bug 1835637 in OpenStack Compute (nova) "(404) NOT_FOUND - failed to perform operation on queue 'notifications.info' in vhost '/nova' due to timeout" [Undecided,Incomplete]
20:04:59 openstackgerrit Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242
20:05:00 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243
20:05:00 openstackgerrit Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707
20:05:01 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
20:05:01 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:07:07 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:07:10 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:44:16 openstackgerrit Lee Yarwood proposed openstack/nova stable/stein: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/696686
20:44:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260
20:45:03 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261
20:49:06 eandersson mnaser I dont't think it's the same issue, but could be similar
20:49:28 mnaser eandersson: seems like the queues just crash even on a restart
20:49:34 eandersson https://github.com/rabbitmq/rabbitmq-server/issues/641
20:49:41 mnaser even with 3.8.1
20:49:44 eandersson This is the issue we are running into
20:50:19 mnaser given that we have k8s running side by side our entire infra now, we're just probably going to run one single non-clustered rabbitmq instance
20:50:23 eandersson but yea could be the same because we see different symptons each time
20:50:26 mnaser for every service
20:51:53 eandersson Yea - this has been very draining for us
20:52:06 eandersson Some of our RabbitMQ clusters has 17k+ queues as well
20:52:27 eandersson Since every compute ends up with like at least 7 queues
20:52:52 eandersson btw mnaser do you have network partition auto-healing enabled?
20:53:11 eandersson or pause_minority rather
20:54:22 eandersson I believe the trigger for these issues is that when RabbitMQ comes back up it is starting to accept connections before it is fully recovered.
20:54:45 eandersson At least in 3.6.X we had that issue. Could be a new issue in 3.7.X
20:55:35 mnaser eandersson: i ran into it even with 3.8.1
20:55:56 mnaser eandersson: cluster_partition_handling, pause_minority
20:56:45 eandersson I tried to tell the RabbitMQ guys about this, but I don't know how to reproduce it properly.
20:56:58 eandersson Plus I run like 3.7.5 so they just keep telling me to upgrade.
20:57:46 eandersson But after the RabbitMQ 3.6.3 debacle we don't upgrade too frequently without first testing it properly. So takes time.
21:00:05 Sundar dansmith: Would it help to discuss in IRC and then summarize in Gerrit?
21:01:14 dansmith Sundar: if you were ever in irc, sure
21:01:56 dansmith Sundar: what I want is discussion instead of replies of "no you're wrong" two minutes before pushing up replacement set without addressing the thing
21:02:45 Sundar dansmith Sure. I addressed most of your points, BTW.
21:03:12 Sundar I think the disconnect is the understanding of the object model. Please see https://review.opendev.org/#/c/631243/46/nova/accelerator/cyborg.py@26
21:04:31 dansmith Sundar: I'm not sure what your point is
21:05:03 dansmith Sundar: I will eventually be able to have one crypto accelerator and one gzip accelerator, right?
21:05:20 Sundar dansmith: Yes, for the same instance
21:05:30 dansmith and those are two device profiles?
21:05:42 Sundar No, one device profile with 2 request groups
21:05:54 dansmith okay, so an instance will only ever have one device profile?
21:06:10 Sundar Yes. That single device profile's name is set in the flavor.
21:06:25 dansmith so, why are you setting the tag in the event to the device profile?
21:07:24 Sundar That was the only logical choice for a tag that seemed relevant.
21:07:59 dansmith setting the tag means that the event needs to be multiplexed for the instance
21:08:22 dansmith which is why we set it to the port id for neutron ports, for example, because there are multiple ports per instance
21:08:54 dansmith I'm not sure why it was decided the encompass all of the accelerators for an instance into a single entity, but alas
21:09:29 Sundar Would we ever need multiple events per instance? For example, for hot adds/deletes in the future, by updating the device profile?
21:09:47 dansmith Sundar: those would be different event types
21:10:47 Sundar Then what exactly is the problem -- that the tag is superfluous?
21:11:08 dansmith setting the tag implies that there can be multiples, so ... yes
21:12:01 Sundar dansmith: Got the disconnect. We don;t use the tag, at least not today. With hot adds/deletes, since it it is going to be another event type, we still don;t need it.
21:12:38 Sundar dansmith: BTW, the idea of one device profile per instance came from the idea of setting one device profile name in the instance.
21:12:42 dansmith I dunno what those would look like, but I would hope that there is some indication of what device the "thing got added" event pertains to
21:13:16 dansmith ...which would be done with a tag
21:14:44 Sundar dansmith: It may make sense to have a single notification for an update too -- because there is not much that Nova can with a partial update knowing that the next event may indicate a failure and things need to be rolled back
21:15:00 Sundar That is the same reasoning as for the bind logic here
21:15:48 dansmith Sundar: "something failed" events are pretty terrible
21:16:33 Sundar dansmith: If you mean that the event should say what exactly failed, agreed.
21:16:55 dansmith Sundar: I mean we should at least know which thing failed, and tag is the "which"
21:17:42 openstackgerrit Lee Yarwood proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909
21:18:57 Sundar dansmith: The problem is that, if I were to send a separate event for each ARQ for binding success/failure, tagged with the ARQ UUID, Nova woudn't know what to do with that . For one, there is no association between the ARQ UUID and the instance in nova -- that would presumably need a db change.
21:19:18 Sundar So, nova wouldn't know which instance is affected.
21:19:48 dansmith huh? the event would still be delivered to the instance, so we know which instance is affected
21:20:22 dansmith I understand we don't store the arq uuid anywhere, although I think we likely still have it before we're going to wait, since we just polled cyborg for the list
21:20:24 Sundar dansmith: true. The event has a server-uuid field.
21:20:35 dansmith that's kindof the whole point.
21:20:58 dansmith it would be much less odd if you did that, even if it doesn't mean that much to us right now
21:21:11 dansmith because we can log the detail, potentially report it in the instance action event,
21:21:28 dansmith instead of "well, I dunno, cyborg said it didn't work.. *shrug*"
21:21:39 Sundar dansmith: However, what does a single ARQ's success mean to Nova? It would have to wait anyway for all of them. If any of them failed, the whole set needs to be rolled back.
21:21:47 dansmith but whatever happens, if you're going to send one per instance, it needs to not have a tag
21:22:12 Sundar dansmith: Ok, I can remove the tag.

Earlier   Later