Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-16
23:22:04 sean-k-mooney but that doesnt make sense
23:22:27 sean-k-mooney the conenct is reestablished at 2019-05-16 18:24:10.109
23:22:43 sean-k-mooney and then there are no log messages untill the time out 1 minut later
23:22:58 melwitt right
23:23:06 sean-k-mooney but the rpc timeout should be longer then 60 seconds right?
23:23:28 melwitt I think it probably defaults to 60s
23:23:40 melwitt I can't remember where it is, lemme see
23:23:55 imacdonn 2019-05-16 18:16:05.973 23144 DEBUG nova.api.openstack.wsgi_app [-] long_rpc_timeout = 1800 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2571
23:24:11 imacdonn wait that's not it
23:24:16 melwitt default to 60 https://github.com/openstack/oslo.messaging/blob/7102eddb1fd799d949ff5dca8aa2637794bb8a43/oslo_messaging/rpc/client.py#L41
23:24:21 sean-k-mooney the long_rpc_timeout is not used for every rpc
23:24:44 melwitt rpc_response_timeout is the one
23:26:25 sean-k-mooney so this is what kombu is doing https://github.com/celery/kombu/blob/master/kombu/connection.py#L500-L547
23:33:56 openstackgerrit Merged openstack/nova master: Fix some issues with the newton release notes https://review.opendev.org/659623
23:34:08 imacdonn attempting repro with kombu debug logging
23:38:45 melwitt good call, because looking at the code it looks like it should be doing the right thing. no clue what's happening
23:42:21 imacdonn weird .. not getting the connection resets now
23:42:41 sean-k-mooney does it not take time for it to happen
23:42:45 sean-k-mooney e.g a few minutes
23:42:54 melwitt did you remember to set threads=8?
23:43:05 imacdonn yeah, I waited 5 mins the first try, then a little over 3 mins second try
23:43:19 imacdonn yes, threads=8, but I should see the connection reset either way
23:43:40 melwitt true
23:45:02 sean-k-mooney ok its technically friday here so im going to have pizza and then sleep. if ye find anything let me know but for no i guess we shoudl update the bug and tell people to run with thread=1
23:45:15 sean-k-mooney our maybe with debug loging...
23:45:31 melwitt pizza, sounds like a solid plan
23:46:24 imacdonn Thanks, sean-k-mooney - pizza/sleep well!
23:46:29 sean-k-mooney i actully had a c++ bug where debug builds did not have the bug because it was an unitialised variable and in debug mode the compiler was "kind" enouch to initalise it to 0
23:48:47 imacdonn ahh, got the reset now ... paste in a few mins
23:51:16 imacdonn http://paste.openstack.org/show/AZQET5IOgntK785RHtxh/
23:57:51 melwitt weird, it says basic_publish_confirm but then one minute later timeout happens
23:58:24 melwitt so confused.
#openstack-nova - 2019-05-17
00:02:03 imacdonn not sure what what basic_publish_confirm message really means
00:02:40 melwitt I don't either but just the indication that it made an attempt to send the message
00:03:00 melwitt I guess that implies it for some reason couldn't get through and results in the timeout?
00:05:33 imacdonn basic_publish_confirm is the method (that's supposed to get autoretry'ed), I guess
00:06:14 imacdonn I think it comes from python-amqp (?)
00:07:31 imacdonn File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1787, in basic_publish_confirm
00:10:59 imacdonn it does seem like it calls it the second time (at 23:48:19.113), but apparently still doesn't get an answer
00:12:07 melwitt yeah, that's the one I was looking at, the second that times out even after re-establishing the connection
00:13:23 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690
00:15:04 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691
00:34:56 openstackgerrit Merged openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374
01:08:39 imacdonn another data-point: the retried request DOES actually get sent, because I see the compute node act upon it ... I guess somehow the client (nova-api) has some issue with the response
03:02:16 openstackgerrit chenker proposed openstack/python-novaclient master: Optimize limit option docs string description for novaclient https://review.opendev.org/643035
03:03:42 openstackgerrit Guo Jingyu proposed openstack/nova master: Skip existing VMs when hosts apply forceconfig_drive https://review.opendev.org/659703
03:17:07 openstackgerrit Guo Jingyu proposed openstack/nova master: Skip existing VMs when hosts apply force_config_drive https://review.opendev.org/659703
04:46:38 ykarel TheJulia, dtantsur|afk can you check comment in https://review.opendev.org/#/c/653279/1
04:48:21 ykarel hmm looks like some issue is already there, as i can see https://review.opendev.org/#/c/659612/1, can you check if this ^^ is same issue or something else
05:25:41 openstackgerrit Guo Jingyu proposed openstack/nova master: Add rfb.VNC support for novncproxy https://review.opendev.org/622336
05:26:26 openstackgerrit Guo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120
06:21:19 openstackgerrit Guo Jingyu proposed openstack/nova master: Skip existing VMs when hosts apply force_config_drive https://review.opendev.org/659703
08:01:14 kashyap aspiers: That's in queue, I'm still backlogged :-( I know it is ready to merge. If I look at, I would like to spend a 2-hour block on it. So trying to find that :P
08:01:18 kashyap aspiers: From your comment:
08:01:19 kashyap "I don't quite understand how the fix in https://www.redhat.com/archives/libvir-list/2019-April/msg01418.html would make the Python arguments optional however, since they're still missing defaults."
08:01:34 kashyap aspiers: Just write to the upstream list (no subscription required; you can ask to be Cced)
08:02:23 kashyap aspiers: But as you noted yourself it's not required, given that you've obtained correct value for @emulatorbin via "Parse <emulator> elements from virConnectGetCapabilities()"
08:03:20 kashyap aspiers: Actually, let me give my review. You've addressed all my concerns
08:24:04 aspiers kashyap: I'm backlogged too ;-)
08:24:55 aspiers kashyap: So I probably owe you some reviews too. Let me know if I can make the review process easier in any way.
08:26:02 kashyap aspiers: Thanks for the offer! I need to do some work similar to yours -- introducing new CPU-related APIs
08:26:55 kashyap aspiers: For this spec: https://opendev.org/openstack/nova-specs/src/branch/master/specs/train/approved/cpu-selection-with-hypervisor-consideration.rst
08:27:01 kashyap See the action items at the end.
08:28:54 aspiers ok
08:45:07 kashyap aspiers: ACKed your change, FWIW
08:45:26 aspiers kashyap: yep, thanks!
08:45:56 kashyap aspiers: I think your change is ready to merge. Let's check with efried (or stephenfin) when they're about.
08:46:14 kashyap aspiers: Hope it won't be delayed, so that you can maintain the momentum.
08:49:04 aspiers Well it's already been delayed a few months ;) but thankfully git makes this much less of an issue. I remember the awful CVS days where the only place for all your pending unmerged work was mushed up together in the work tree, and you had to untangle each new commit-to-be from that
08:49:17 aspiers Drove me crazy
08:56:00 jangutter aspiers: The absolute _best_ thing about git is when you learn about rebasing. And the nice thing about gerrit is it fills in the gaps (tracking iterations of the same "patch queue").
08:56:41 aspiers jangutter: agreed :) git has lots of "best" things though, e.g. I remember when I learnt about the reflog it blew my mind
08:56:58 aspiers ditto git-rerere
08:57:10 aspiers so many cool tricks
08:58:05 jangutter aspiers: who knew that a content-addressable file system accidentally solved the "version control" issue :-p
08:58:27 aspiers :D
10:15:57 kashyap aspiers: Yeah, I see what you mean. I didn't have the misfortune to use CVS, but I did start with SVN, though
10:16:08 aspiers I started with RCS :-o
10:34:59 kashyap Hope you've fully recovered
10:35:26 aspiers LOL
10:35:43 aspiers Actually RCS can be surprisingly effective for particular use cases
10:35:54 aspiers but yeah, it's very limited
10:36:35 aspiers I actually talked a bit about the evolutionary history of version control systems in this interview http://episodes.gitminutes.com/2015/03/gitminutes-32-adam-spiers-on-git-deps.html
10:37:27 aspiers according to the index that starts around 07:08
10:52:55 cdent What are the options for a VM after it is evacuated? I ask in relation to this new bug: https://bugs.launchpad.net/nova/+bug/1829479
10:52:57 openstack Launchpad bug 1829479 in OpenStack Compute (nova) "The allocation table has residual records when instance is evacuated and the source physical node is removed" [Undecided,New]
11:20:17 sean-k-mooney cdent: in what sense
11:20:47 sean-k-mooney cdent: if it is evacuated sucessfully its as if ti was migrated
11:21:10 sean-k-mooney cdent: assuming it was on shared storage its state was not even lost
11:21:25 sean-k-mooney cdent: althogh if it was on epheraml stoage it wasd effectivly a new instance
11:21:32 cdent sean-k-mooney: So evacuate means "recreate this vm somewhere else"?
11:21:39 sean-k-mooney cdent: yes
11:21:53 sean-k-mooney but keep the same ips and volumes
11:22:00 cdent So shouldn't that mean that when the evacuate finishes an allocation is made which replaces the old allocations?
11:22:06 artom Rebuild on a different host, basically
11:22:16 artom So yeah, allocations need to move
11:22:19 sean-k-mooney so if its boot from volume or you hapeend to have the rbd image backend you keep you root disk
11:22:33 aspiers PLEASE can we use the word "resurrect" instead of evacuate wherever possible
11:22:42 sean-k-mooney aspiers: no
11:22:50 sean-k-mooney :P

Earlier   Later