Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-11
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:05:02 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:05:03 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:07:07 openstackgerrit Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245
20:07:08 openstackgerrit Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Enable hard reboot with accelerators. https://review.opendev.org/697940
20:07:09 openstackgerrit Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735
20:07:10 openstackgerrit Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999
20:44:16 openstackgerrit Lee Yarwood proposed openstack/nova stable/stein: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/696686
20:44:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697260
20:45:03 openstackgerrit Lee Yarwood proposed openstack/nova stable/queens: block_device: Copy original volume_type when missing for snapshot based volumes https://review.opendev.org/697261
20:49:06 eandersson mnaser I dont't think it's the same issue, but could be similar
20:49:28 mnaser eandersson: seems like the queues just crash even on a restart
20:49:34 eandersson https://github.com/rabbitmq/rabbitmq-server/issues/641
20:49:41 mnaser even with 3.8.1
20:49:44 eandersson This is the issue we are running into
20:50:19 mnaser given that we have k8s running side by side our entire infra now, we're just probably going to run one single non-clustered rabbitmq instance
20:50:23 eandersson but yea could be the same because we see different symptons each time
20:50:26 mnaser for every service
20:51:53 eandersson Yea - this has been very draining for us
20:52:06 eandersson Some of our RabbitMQ clusters has 17k+ queues as well
20:52:27 eandersson Since every compute ends up with like at least 7 queues
20:52:52 eandersson btw mnaser do you have network partition auto-healing enabled?
20:53:11 eandersson or pause_minority rather
20:54:22 eandersson I believe the trigger for these issues is that when RabbitMQ comes back up it is starting to accept connections before it is fully recovered.
20:54:45 eandersson At least in 3.6.X we had that issue. Could be a new issue in 3.7.X
20:55:35 mnaser eandersson: i ran into it even with 3.8.1
20:55:56 mnaser eandersson: cluster_partition_handling, pause_minority
20:56:45 eandersson I tried to tell the RabbitMQ guys about this, but I don't know how to reproduce it properly.
20:56:58 eandersson Plus I run like 3.7.5 so they just keep telling me to upgrade.
20:57:46 eandersson But after the RabbitMQ 3.6.3 debacle we don't upgrade too frequently without first testing it properly. So takes time.
21:00:05 Sundar dansmith: Would it help to discuss in IRC and then summarize in Gerrit?
21:01:14 dansmith Sundar: if you were ever in irc, sure
21:01:56 dansmith Sundar: what I want is discussion instead of replies of "no you're wrong" two minutes before pushing up replacement set without addressing the thing
21:02:45 Sundar dansmith Sure. I addressed most of your points, BTW.
21:03:12 Sundar I think the disconnect is the understanding of the object model. Please see https://review.opendev.org/#/c/631243/46/nova/accelerator/cyborg.py@26
21:04:31 dansmith Sundar: I'm not sure what your point is
21:05:03 dansmith Sundar: I will eventually be able to have one crypto accelerator and one gzip accelerator, right?
21:05:20 Sundar dansmith: Yes, for the same instance
21:05:30 dansmith and those are two device profiles?
21:05:42 Sundar No, one device profile with 2 request groups
21:05:54 dansmith okay, so an instance will only ever have one device profile?
21:06:10 Sundar Yes. That single device profile's name is set in the flavor.
21:06:25 dansmith so, why are you setting the tag in the event to the device profile?
21:07:24 Sundar That was the only logical choice for a tag that seemed relevant.
21:07:59 dansmith setting the tag means that the event needs to be multiplexed for the instance
21:08:22 dansmith which is why we set it to the port id for neutron ports, for example, because there are multiple ports per instance
21:08:54 dansmith I'm not sure why it was decided the encompass all of the accelerators for an instance into a single entity, but alas
21:09:29 Sundar Would we ever need multiple events per instance? For example, for hot adds/deletes in the future, by updating the device profile?
21:09:47 dansmith Sundar: those would be different event types
21:10:47 Sundar Then what exactly is the problem -- that the tag is superfluous?
21:11:08 dansmith setting the tag implies that there can be multiples, so ... yes
21:12:01 Sundar dansmith: Got the disconnect. We don;t use the tag, at least not today. With hot adds/deletes, since it it is going to be another event type, we still don;t need it.
21:12:38 Sundar dansmith: BTW, the idea of one device profile per instance came from the idea of setting one device profile name in the instance.
21:12:42 dansmith I dunno what those would look like, but I would hope that there is some indication of what device the "thing got added" event pertains to
21:13:16 dansmith ...which would be done with a tag
21:14:44 Sundar dansmith: It may make sense to have a single notification for an update too -- because there is not much that Nova can with a partial update knowing that the next event may indicate a failure and things need to be rolled back
21:15:00 Sundar That is the same reasoning as for the bind logic here
21:15:48 dansmith Sundar: "something failed" events are pretty terrible
21:16:33 Sundar dansmith: If you mean that the event should say what exactly failed, agreed.
21:16:55 dansmith Sundar: I mean we should at least know which thing failed, and tag is the "which"
21:17:42 openstackgerrit Lee Yarwood proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909
21:18:57 Sundar dansmith: The problem is that, if I were to send a separate event for each ARQ for binding success/failure, tagged with the ARQ UUID, Nova woudn't know what to do with that . For one, there is no association between the ARQ UUID and the instance in nova -- that would presumably need a db change.
21:19:18 Sundar So, nova wouldn't know which instance is affected.
21:19:48 dansmith huh? the event would still be delivered to the instance, so we know which instance is affected
21:20:22 dansmith I understand we don't store the arq uuid anywhere, although I think we likely still have it before we're going to wait, since we just polled cyborg for the list
21:20:24 Sundar dansmith: true. The event has a server-uuid field.
21:20:35 dansmith that's kindof the whole point.
21:20:58 dansmith it would be much less odd if you did that, even if it doesn't mean that much to us right now
21:21:11 dansmith because we can log the detail, potentially report it in the instance action event,
21:21:28 dansmith instead of "well, I dunno, cyborg said it didn't work.. *shrug*"
21:21:39 Sundar dansmith: However, what does a single ARQ's success mean to Nova? It would have to wait anyway for all of them. If any of them failed, the whole set needs to be rolled back.
21:21:47 dansmith but whatever happens, if you're going to send one per instance, it needs to not have a tag
21:22:12 Sundar dansmith: Ok, I can remove the tag.
21:22:22 dansmith Sundar: in some future we may need to do extra things to the device on the host, and we can do that for devices that complete early while we wait for the others,
21:22:40 dansmith and like I said, being able to report to the user and the admin which and what happened is useful
21:24:38 Sundar Re. report to user/admin, the exact details would be in Cyborg logs. Given the range of errors, it will be tough to state the failure reason in the Nova event.
21:24:52 Sundar Perhaps the Nova logs could point the admin to Cyborg logs.
21:25:12 dansmith no, I mean report *which* thing failed
21:25:13 dansmith not why
21:26:42 Sundar For example, say we had one event per ARQ, tagged by ARQ UUID. Nova logs report that a certain ARQ UUID failed to bind, Would that really be useful, without knowing more about the ARQ?
21:26:53 dansmith yes?
21:27:22 dansmith I grab that arq uuid and hit logstash to see what else has reported stuff about that thing
21:28:27 dansmith We could just always report "error with (neutron|cinder|cyborg). You go figure it out" but that's not likely to be very popular
21:30:11 Sundar dansmith: IMHO, I think it will be better to look at an error in context. Giving the user an ARQ UUID is equivalent to 'go search Cyborg logs', right?
21:30:33 dansmith I can't believe I'm having this conversation.
21:30:48 dansmith Sundar: either make the event look like the rest of our events, or stop passing the tag.
21:32:17 Sundar dansmith: Sure, I said I can stop passing the tag. You had 2 concerns with that -- one was ease of use of logs. We were attempting to resolve that.
21:32:51 Sundar The other was "in some future we may need to do extra things to the device on the host".
21:33:35 Sundar I am still thinking about that, esp. if those extra things will require a different event type, which could have or not have a tag, independent of this event.
21:34:05 Sundar dansmith: Please LMK if I am off.
21:35:51 dansmith Sundar: is there any reason you can't send the event per ARQ in cyborg, and any reason you can't know the ARQ UUIDs at the time we wait? We have that info passed in now right?
21:38:58 Sundar dansmith: We are debating the benefits of sending one event per ARQ, beyond 'this is what we have done in the past.' It seems to me that logging the ARQ UUID gives little info to the user beyond 'look elsewhere for the details'. Please LMk wht that is wrong.
21:39:32 Sundar Secondly, we still have to wait for all events for that instance, right?
21:39:57 dansmith efried: you around now?
21:40:11 efried hi, yes, all the doctors this week.
21:40:23 efried shall I read scrollback or are you going to summarize for me?
21:40:40 dansmith so, it has become clear to me that the cyborg event as currently proposed is being used very differently than what we have for the other services

Earlier   Later